Not long ago I attended Automation Guild Conference. There I’ve learned about the possibilities to integrate visual testing in the existing automation framework. The theme was interesting to me, because it has answered the question regarding the mobile automation usefulness. While this conference the presenters used Applitools for their visual tests. So, I made up my mind to give it a try.
Why visual testing is a great supplement to the automation
For a long time I’ve heard things like: “automation is useless” or “the automation results is not something you can trust”. There are some arguments to this opinion. Firstly, the automation tests the functionality without checking how the screen looks like. That means that the automation doesn’t care where the elements are and there is a possibility to touch elements that are invisible for a user, though the test will still pass. Secondly, not everyone trusts the automation, as they don’t believe in green builds on the CI. While the second point looks to me as a gap in the technical background, the first statement sounds valid to me. The visual testing is able to answer the both points. Using additional visual checks in the automation you verify not only the functionality but also the user flows. It is visible on all the levels, because screenshots of the test flow is something that you can easily present to non-technical people and the visual checks will take care about the possible design issues.
What it means for an existing project
To be honest, yet I don’t know how the setup will behave for the whole project. What I have already tried is to pick up a small test suite and integrate Applitools there. All the process took not more than 15 minutes. I use Calabash for our automation. There is no integration between Applitools and Calabash, but I took Applitools’ java solution to upload images to the Cloud. The Java solution doesn’t cover 100% of our needs but it is good enough to create a proof of concepts. Applitools has integration with Selenium and Appium, so if you use one of these frameworks, everything should be even easier. In addition, Applitools is able to do screenshots of the whole screen/page when using Appium/Selenium. If you use another framework like I do, it is also not a problem. Applitools have SDKs that are not locked by any technologies. In the future we’re planning to use their Ruby SDK.
So, let’s get back to the project. You will have to find a good way to do the proper visual checks. In our project we have flags that identify transition from one screen to another. I can enable visual checks in this place and they will be immediately applied to the whole project. Codewise, it will not take longer than a minute, but.. I’m really not sure how many hundreds of failures we will get after enabling it for the whole project. The main problem that I can see now is preparing tests to be testable visually. I can already name 10-20 places in our APP where it will fail, because we have dynamic data there. I will still have to figure out how to prepare all the tests in order to get the same views every time.The thing that could help is different levels of comparisons provided by Applitools. It is possible to deselect parts of the screen that you want to ignore or you can change the check from ‘one to one’ comparison to ‘Content’ or ‘Layout’, if it passes to your scenario. I have a positive feeling about this challenge. I believe that I will be able to convert the biggest part of the tests very fast.
Full integration or just a visual testing
The complete integration of automation tests with visual checks looks just awesome. The overall testing should be more precise, so you will be able to get rid of the assertions that will not be needed anymore. Applitools will be able to take care of almost all the assertions. Also it will be possible to start the automation of localization and that is also a very good point for us. So in fact, if Applitools can be used like a general solution, it will make your project simpler and easier to maintain. But how I’ve already said, the only question will be if there is a possibility to use visual checks in the existing tests.
There is also a possibility of using Applitools as a separate set of tests that will cover only visual tests. This will also be very useful for the project, but I prefer the idea of full integration. Though, partly integration is a way better than nothing.
Sync with designers
Yeah, this is an amazing opportunity to sync with your designers. I often hear that mobile designers don’t see the product they have designed until the APP is realised. The reason is that they will need to build the APP every time they want to see their components in combination with other components in the APP. And mostly designers don’t have developer tools on their machines for doing that. Also there is a communication problem, as designers have to know on which point the feature was pushed to the master. So these things are really easy to forget. The visual testing automatically resolve these problems:
- Designers have access to all the recorded tests
- Designers will automatically get all the failures after the nightly build and can see if something went wrong with the new components
We are still in the evaluation process of Applitools. So far I can say that it looks very stale and promising. Recording and re-recording of the tests are really. That is quite a big plus, because from other tools I know that it is the main pain point. I will return to this topic another article when our evaluation is done. There I will share the experience of working with Applitools engineers and the platform in general.