An almost perfect Test Suite

Matthias Kempka
September 9, 2010 | 3 min Read

During RCP application development the creation and maintenance of a Test Suite is a common annoyance. While solutions exist that we can live with, the current state of test suites is ennoyance enough that it was a topic for a talk at the Eclipse Testing day recently. Further down I’m presenting a way to create Test Suites that overcome most problems of the presented methods. But before that, I’ll discuss a bit background knowledge.

Background

Good test suites

  • are easy to set up
  • pick up new tests as they are included in the projects (no need to include them manually)
  • can differentiate between long running tests and fast unit tests
  • Can be used in both the IDE and the continuous integration
  • Don’t add performance

One way to create dynamic test suites is the ClassPathSuite. It goes through the classpath and adds all tests it can find to a test suite which is then executed. This approach has two problems that both come from loading all classes before inspecting them:

  • The time to dynamically create the test suite goes up
  • Static initializers may have wrong assumptions

For OSGi applications there is another way to create a test suite dynamically, thats the BundleTestCollector from Patrick Paulin. It is a similar approach in that it goes through specified bundles and adds classes to a test suite, but has neither of the above drawbacks

  • No class is loaded before a test is actually executed
  • There is no need to scan dependencies: You can narrow down the plug-ins that are scanned for tests

Sounds almost perfect, doesn’t it? Hold your breath, it has some problems, too, but they are small against everything else I stumbled across.

First, Patricks version only works with JUnit 3. I used a modified version for a while now that works with JUnit 4 and has some more assertions than Patricks original version. The remaining problem is, when looking at the test results in the JUnit view, the stack trace is not linked to the workspace. A double-click fails, and I need to open the class and navigate to the line by hand.

The BundleTestCollector requires OSGi Bundles. You can’t just run it as JUnit test. In your IDE it must be started as PDE Test (or SWTBotTest), and in the continuous integration, you should use the Eclipse Testing Framework or the SWTBot test runners.

The BundleTestCollector

Patrick published his version of the BundleTestCollector under the EPL, so my modifications are EPL, too. Here is a zip containing the BundleTestCollector and it’s single dependency, an Activator. You’ll have to copy them into an actual bundle and adjust the namespace. JUnit4 must be on the bundle dependency path.

The BundleTestCollector picks up the tests by naming conventions. If your test doesn’t adhere to a naming convention, it won’t be picked up.

Here is how to define a TestSuite. It’s still a JUnit 3 test suite, but the actual tests may be both, JUnit 3 or JUnit 4.


public class MyTestSuite extends TestSuite {

public static Test suite() { BundleTestCollector testCollector = new BundleTestCollector(); TestSuite suite = new TestSuite( "All Tests" ); testCollector.collectTests(suite, "my.bundle.namespace", "my.package.namespace", "\*Test"); // add more lines collectTests(...) calls if necessary return suite; } }

Hope that eases some of your test suite maintenance pains.

Matthias Kempka