Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight)

Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight)

For the last few years I´ve programmed mostly in Java. And, in this world it´s common to have a Continuous Integration (CI) server for building your product nightly or more often. When we started developing RAP mobile with its large amount of Objective-C code within an Xcode project, we didn´t have a CI server for the iOS client. But this has changed recently. We now have a full CI system for our iOS products including automated test execution, code coverage and publishing to Testflight. In this post I´d like to show you how we created this system as it may be helpful for you when you´re building your own app.

First I´d like to mention some resources that helped me a lot. These are blog posts from Shaun Ervine, Lars Rückemann (German) and Dustin Mallory. Thanks to these guys I was able to get started with the CI for iOS. One disadvantage of the resources is that they are mostly step by step tutorials or were written for older Xcode versions and were not helpful with some of the problems you´ll encounter when setting things up. I will go in detail shortly, but let´s get started with the easy part/

Setting up a Jenkins slave

Here at EclipseSource we use Jenkins for our build jobs. It runs on a simple Linux machine and is all we need for our Java projects. But for an Xcode project you need a Mac to build it. Luckily Jenkins offers the option to hook other Jenkins instances as slaves. So it made sense to hook a Jenkins Mac slave to our “Master” and build all Xcode jobs on it. Renting a Mac is not so easy/cheap as it is with Linux machines and I found only one interesting offering from macincloud which in the end, didn´t fit our needs. So, we bought a Mac and hosted it ourselves. The cool thing with Jenkins slaves is that they don´t need to be available for the outside world so you can simply hook the slave via JNLP. Our setup on the master is shown in the screenshot below. You´ll find a very detailed tutorial on the Jenkins Wiki on setting up the slave. The most important thing for me was to restrict the usage to tied jobs only because I didn´t want our Java projects to be built on the slave too.

jenkins slave Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight)

After configuring the master it was time to start the Jenkins slave. The cool thing about this is that you don´t need to install a full Jenkins on the slave. All you need is a simple jar file called slave.jar, that you can download directly from your Jenkins instance. To start the slave I wrote a short shell script called startSlave.sh which starts the .jar file as described here. One problem I encountered was the OSX launchctl concept, which was intended to be a better initd. Yep, you read it correctly, I said “intended to be”. Writing launchctl scripts is a mess. In the end, I wasn´t able to write a launchctl script to start the slave without logging in as the user. So, as a workaround, I called the startSlave.sh with launchctl from an ssh session. This does not ensure the restart of the slave on a system restart, but it does restart it if problems occur during uptime. If anyone has a working script I would be more than happy to see it in a comment icon wink Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight) . Calling the startSlave.sh script looks like this:

launchctl submit -l jenkins-slave -- sh /Users/build/scripts/startSlave.sh

Of course you don´t need the startSlave.sh script. You can also run the slave directly by launching the jar.

Once we had a running Jenkins it was time to get the build tools on it. This basically means Xcode which can be installed via the Appstore (at the time of writing Xcode 4.3.2 is the latest). After installing Xcode I played around with a few initial iOS jobs and quickly ran into a new problem.

Building Xcode projects from the command line is the same as building it headless using Jenkins. The first thing that happened was that I got an error saying “.. No developer directory found at /Developer”. The reason is that by default, with Xcode 4, the developer directory changes from /Developers to /Applications/Xcode.app/Contents/Developer/. The commandline tool xcodebuild needs a valid location for this directory which is not set up by default. To fix this you can use the xcode-select command line tool and the following command:

sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer/

After doing this we had everything we needed for creating our first iOS job on the Jenkins Master.

Creating a build job

Before we can build Xcode projects in our Jenkins we need to install a Jenkins plugin called “Xcode Plugin.” This plugin makes your life much easier when building Xcode projects. After installing it, I created a freestyle project and added an Xcode build step. It was pretty straight forward to configure it as described on the plugin´s site. The more interesting part sits in the targets of our Xcode project. As I mentioned earlier I wanted to build an ipa and execute tests automatically. Basically this means I have two targets, one for the app and one for the tests. How to configure these targets is described in several other blog posts. I can recommend this one from Software Noise and its follow ups. After configuring the targets, I faced two problems. The first one is that my tests weren´t executed, which is a bad thing for automated tests icon wink Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight) . The reason for this was that every blog I read said that I should disable “test after build” in the target. Hmm, let´s think about this: d.i.s.a.b.l.e “test after build”. I think it´s obvious why the tests weren´t executed. After enabling it the tests ran. Pretty much immediately I encountered another problem and this was a hard one. I have to add here, that from my point of view the solution is anything but clean.

When running Xcode tests headless you can´t have a “Test Host” in your unit test target. The xcodebuild command line tool fails on executing the tests and prints out this error:

Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).

After a long search for a solution I found a post on StackOverflow. Basically it says that you need to patch a file on the Jenkins slave. Trust me, this was the last thing I wanted to do, but after wasting a whole day on this problem it was the only way out. So, patching the script as described in the stackoverflow post worked out pretty well. My tests were all executed and green. Yippie icon wink Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight)

One of the coolest things about the Xcode plugin is that it converts the OCUnit test reports directly to a format the Jenkins JUnit plugin can understand. So, to see our test reports on Jenkins all I needed to do was publish the reports as in the screenshot below.

test reports Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight)

 

So far so good, we had running tests and were able to build an ipa file. Now it was time for code coverage.

Cover the code

Xcode has built-in functionality to record coverage reports. How to set this up is described in detail in this blog post. To get this to work for me, I had to enable “Generate Test Coverage Files” and “Instrument Program Flow” in the target of my app too. After doing this the report files (.gcda, .gcno) were generated correctly. This is pretty cool but sadly Jenkins has no plugins for reading these reports. After searching a bit, I found a solution that´s almost sexy. There is a script for converting the coverage reports into cobertura coverage reports which can be read by the Jenkins cobertura plugin. This script is called gcovr and can be downloaded here. I copied it into our git repository and executed it in another build step by running a simple shell command:

cd ${WORKSPACE}
 ./scripts/gcovr -r ${XCODE_PROJ_FOLDER}/ -e '.*/*Test.*' -x > coverage.xml

The syntax of the script params is a bit odd but it works. The setup for collecting the cobertura reports is shown in this screenshot:

cobertura Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight)

After setting this up our build job publishes reports that are quite good icon wink Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight) . We then had automated tests, code coverage reports and a working .ipa file. It was time to publish it.

Publishing to Testflight

For those of you who don´t know what testflight is, take a look at their website for more information. To sum it up, it´s currently the most convenient way to publish iOS apps without publishing them on the Appstore. What I needed was automated publishing to testflight to enable our users to download the latest version continuously. The cool thing is that this was the easiest task. There is a Jenkins plugin called the “Testflight Plugin” which basically serves the REST API from Testflight. All I did was create a separate job for publishing. In this job I used the “Copy Artifacts Plugin” to copy the .ipa file from the other job and published it via the Testflight plugin. It´s that easy. The configuration can be similar to the one in the screenshot below.

testflight Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight)

Conclusion

As I mentioned in the beginning of this post, most of the time I program in Java. In the Java world the CI topic is pretty mature and you can find plugins, scripts etc. for nearly every problem you run into. When doing CI with Objective-C projects this is different. I think the coolest concept are the targets of Xcode projects. Everything you need to configure you can do within the target. The thing that is lacking is the integration into CI systems like Jenkins. It would be great to see this change in future when more Objective-C specific plugins are developed. Meantime, I hope the solutions for the pitfalls we encountered helps you setting up your environment. Please feel free to add more pitfalls/solutions in the comments.

3 Comments
  • Marco Massenzio
    Posted at 20:14, 2012-06-06

    Talking of code coverage, it’s my understanding that neither Emma nor Cobertura work for Java 7 – is this correct?
    if not, what am I missing, and if yes, any `top tips` as to any tool that does?

    I’m a fanatic of CI, testing and code coverage, and I’m not quite ready yet to give up on that – but this is blocking us from moving to Java 7 and that’s not good either…

    Thanks!

  • Pingback: Ups and Downs with Continuous Integration for iOS Apps (Jenkins, Xcode, Cobertura and Testflight) — Graine de kawa.
    Posted at 20:17, 2012-06-06