In this article I will describe the way of running jobs on CI from the comment in GitHub pull request. This can be any kind of job you may need to perform, but mostly it makes sense to run tests there. At XING, we have successfully implemented it and have the possibility to run Unit tests, Automation tests, Visual tests and static code analyzers from the pool request.
How it looks like
In the screenshot below you will see an example of executing Calabash tests using the tag parameter.
So, after leaving a comment with the test phrase:
please test Calabash <YOUR PARAMETER>, the job with the given parameter will be triggered. This job will build the APP from the branch of current PR and start tests against this build. Some minutes later you will get a nice report with your test execution results. In the example above you can see a passed test run.
In the result of failed test run you can see which tests were failed. Here is an example of failed suite:
Later in this article I will describe how to configure info you want to reflect in your report.
Note: This is applicable not only for test automation. In the same way you can configure your unit or any other kind of tests.
Why you may want to run parameterized builds
There are a few scenarios where you can use this setup, but most problematic aspects are time and resources. So, either you have too many tests and it is time consuming to run all of them on every change on repository, or you have a lot of pushes per day and your infrastructure can’t handle it, because you don’t have enough hardware.
I think that this setup really suites to test automation. As you may know, test automation on mobile is very slow and you can’t just run all the tests on every single commit. Moreover, I see a really good use case here. Imagine that you’re refactoring one of the sections in your APP. You think that everything is fine, but you want to make sure that you haven’t broken anything before merging the PR on master. That’s where this setup can help speed up the development. You can run the whole suite of tests for this section and the testing will be completed even before PR will go through review. So, you can merge your PR faster, without a lot of manual testing efforts.
To get all this working you will have to use some Jenkins plugins and Ruby gems.
This plugin will do the test phrase recognition and job running. When you install it, you will see a new checkbox in the ‘Build Triggers’ section. Clicking on it, you will be asked to provide your GitHub API credentials and list of admins that are able to trigger a job.
In the same field you will see ‘Advanced…’ button. In the screenshot below you will see a configuration for my Calabash test phrase.
Here you can configure your test trigger phrase. You can use there a string or a regular expression. In my case it is an expression that will take everything that contains
please test Calabash as a test phrase and will start the test run.
Another button that you will have in this plugin is ‘Trigger Setup…’. Here you can configure the status and success/failure messages. I’m using here the comment from GitHub in order to see which parameter was passed to the test. Here is the configuration:
Also, do not forget to get the right branch. The plugin has an environment variable with the actual commit name. Here how it is configured in the Git repositories configuration:
- setting an order for the post-build actions. It is not really important when you already have JUnit XMLs, but you will need it if you have to generate them.
- allowing any step as a post-build action, including the “execute shell” step. Without the plugin, you can’t run ‘danger’ as a post-build step.
- preparing and publishing your JUnit report.
Both plugins #2 and #3 you will combine and put in post action:
If your tests generate JUnit reports, you can publish the report right away, as it is done on the screenshot above. If there is no JUnit reports, you can add ‘Execute shell’ step before Publishing and generate the report. For iOS Unit Tests we use trainer gem. For Android you can use gradle, more about this you can read in this article.
When you have the JUnit report, you may want to push it to your PR to have a nice report that you have seen in the very first screenshot in this article. To achieve that you can use danger. In ‘Execute shell’ step you will push a new comment to GitHub using your GitHub token and host.
Getting parameters from your GitHub comment
In case you run the job from GitHub comment, Jenkins job will see an environment variable $ghprbCommentBody. This variable contains your test phrase that you can parse in order to get the parameter for test run. In case with Calabash tests, I will need to get the tag in order to run tests. I will parse the GitHub comment right before running the test.
Here is an example of getting everything after @ sign:
Using regular expressions you will be able to get any parameter you want.
Danger output is configured in the project. Configuration is stored in
Dangerfile that is placed in main directory. Here is an example of
Dangerfile. On the moment when
Dangerfile is executed, you will have access to all the Jenkins parameters* and your JUnit reports. Using this data you can add some meaningful information to your output in PR.
*Don’t mix Jenkins parameters with environment variables. To get environment variables visible, you will have to inject them into parameters using Environment Injector Plugin
Note: Don’t forget to check the “Execute concurrent builds if necessary” in your Jenkins job. It will be needed for the whole setup.
This is all I wanted to share in this article. If there are any open questions, just leave a comment below.