In this article I’m going to describe several strategies on how frequently you should run automation in your project. I will try to tackle all the possible variations, so everyone will be able to pick something out that suites for his/her project.
You can ask, why are there more than one possible solution? The answer is simple: it depends on several factors. All the companies, that are using automation in their projects, have different needs and financial situations. They are using different development methodologies, various technologies and ways of working. For example, the approach, that successfully works in a big company, will unlikely work for a small one.
From my experience, nightly builds is a common solution for the most of the projects I’ve seen. Why? Simply because it covers more or less a bigger part of what every company needs from the test automation, also it is the simplest technical solution, that will not require a lot of resources. So, you will not have to invent a bicycle but use a ready solution instead.
Let’s try to understand what nightly builds can cover: every morning it gives you a good overview on what has been done in your APP the previous day. If something is broken, developers can still fix it quite fast, because they finished their work on the functionality, that broke tests, not so long time ago. And, if everything runs successfully, you have more confidence in the current release state.
This approach is really good already, but later in this article I will show why it can’t be enough for some companies.
The first reason to choose the automation only for release builds can be a small APP under test and a believe in manual tests. When you have this setup, it can be enough for you to run automation just against release candidates.
The second reason to choose this model is opposite situation, when your automation is huge, and it takes multiple days to finish test runs and analyze the results. I worked in a such environment and strongly believe that, when your automation takes too long time to run, then something is completely wrong. But let’s imagine, that you have some system limitations or not enough computing resources. So, in this case running automation only for releases will not really suite, but it is better than nothing.
This approach will not differ too much from the release builds approach. But, if release builds approach supposes to have some automatic process in place, the manual one is just about running tests manually, when you think that it is the right time to do it. This will work for very small projects with rare releases.
Running automation on pool request (PR) basis
This is the part, where I will explain why nightly builds are not enough for some companies, including the company where I currently work. Why do we have special requirements and why running nightly builds doesn’t work for us anymore?
One of the reasons is that we have quite a big amount of teams and developers, that are working on the same APP, and this creates some problems for the automation.
- Automation becomes fragile. After 10-20 merges per day it can be quite hard to understand, what exactly was the reason for tests to break. Everyone, whose section is broken, starts to look at the automation and looses additional time, because the breaking change could be done by another team.
- Developers tend to merge everything in the last moment (just before the code freeze). This has become a real problem, because every code freeze provokes a lot of work to do in order to fix the tests before the release testing starts.
So, after catching some release bugs, that could have been found earlier and being busy for almost the whole code freeze day, we decided to try a more complex solution and spread this work to the PRs, to where these bugs and broken tests belong. As a main advantage I see the possibility to have on master branch only tested code.
In one of my previous articles I have already explained how we run automation tests from the pool request on GitHub. If you haven’t read it, just take into account, that GitHub has the possibility to run builds on Jenkins, using comments in the PR against APP that is based on the PRs’ branch.
So, in our setup every Dev or Test Engineer can run automation right from the PR and get the report back. This appears to be very efficient, because currently every Dev can see, how he/she has broken the test or the APP functionality, and fix it immediately in the same PR. This has really helped to save the time, that was spent on analyzing test reports and finding out who and how can fix the issue. And as a final measurement, we are noticing less release bugs than before.
The main constraints here are the CI resources. Ideally I would like to run all the automation tests automatically and not just on every PR, but also on every commit. But as you may already know, mobile automation is very time consuming, and in order to achieve running it on every commit, you will need to have a really powerful CI system.
I tried to touch the problem from the different angles and give you an overview of the problems which you can have or would probably have in the future and how we solved them in our company.