The general theme for this year’s testing day is “Testing and Beyond”. Topics of interest include:

1. Testing Eclipse applications
2. Testing within the Eclipse Ecosystem
3. Testing on Eclipse Projects
4. Design for testability in Eclipse
5. Case studies of testing projects
6. Eclipse tooling and technology for the test process
7. Testing as a part of the application lifecycle
8. Continuous integration and testing for Eclipse applications

The call for papers is open. If you would like give a talk on one of the given topics, or maybe if you have a visionary ‘beyond’, don’t hesitate to submit a talk.  The deadline for submissions is May 31st.

You’ll find more information on the wiki at http://wiki.eclipse.org/Eclipse_Testing_Day_2012

In the morning we heard several talks about various testing strategies in different commercial products. Alexander Klein from BeOne held an inspiring talk about testing the users experience for a product. Among other ideas he recommended to watch users while they are confronted with the product to get ideas for the next iteration.

In the afternoon things became more hands-on. Michaela Greiler presented her Eclipse Plug-in Test Suite. This is a test runner that allows static and dynamic analysis to report on executed extension points and OSGi services. By itself it is merely interesting, but imho not interesting enough to be a standalone project. I think this technology should go as additional metrics in something like EclEmma. Maybe JaCoCo will provide a home? Hear me, Marc?

Jubula was a major point of talk. Felix Ziesel from BREDEX showed how he uses the Eclipse for Testers download to define an automated smoke test suite to test the completeness of the Eclipse for RCP and RAP developers. I hope he will repeat this talk at an EclipseCon where other packagers are present. This just looked as it ought to be, but obviously it wasn’t feasible before Jubula.

The last talk was mine. I presented an example of a full-featured PDE build from a git repository with test automation. Even though it was the last one of a full day of talks the discussion afterwards was interesting and showed I had hit a sore spot in many projects.

At the side where a few booths from various companies. I was excited to see Xored advertising Q7 there. This is a UI testing tool that only supports Eclipse. Because of this exclusiveness the scripting language gets right to the point. Capture and Replay functionality is also available. Why was I excited about that? – Coming as complete outsiders, they did a talk at last years Eclipse Testing Day about this side product they developed to make testing their application easier. It was extremely well perceived and for a while they where the focus of attention with this thing that’s now Q7. I’d really like to see them more involved with the Eclipse community. Specifically I think their tool might be in a good position to allow UI testing for a RAP workbench.

As a conclusion I have mixed feelings about the Eclipse Testing Day. It shows that things are moving and keeps a handful of people talking. I think it could be a real benefit for the community, if it would show up. But while last year all talks where held in english, now only 3 out of 10 did so, making the Testing Day too much of a german and too less an international event. Many key players where missing. Nobody mentioned SWTBot or JaCoCo. WindowTester didn’t even seem to exist. Is FrogLogic still in business (Their website says, yes)?
It seemed to me that compared to last year there was almost an entirely new audience. I recognized only a handful people and 2 of them where speakers.
It think that the talks and the audience are a bit too diverse. The name “Testing Day” brings together developers people who drive test processes in larger companies, but did’t lead them to talk to one another. Instead of accepting just mixing them in a single branch of talks, the Testing Day could improve by cutting down the program for the whole audience to maybe half a day. Fill the other half day with special interest topics, i.e. bring together the developers to talk about TDD, JUnit, issues about test automation with their test runners and hard times with AspectJ. Bring the people together who want to discuss test processes in big projects, but separate these two groups for a while at least.

The organizers are aware of this, too, so I’m looking forward for a strong and interesting Eclipse Testing Day 2012.

I’m planning a series of blog entries with and around that example. Some features are:

• Works system independent
• Executes tests
• Builds from map files
• Separates compile step and assemble step (Improves build time)
• Mostly self-contained (Required: An Eclipse SDK and Git)

How to get it running in short:

• Fetch from git://github.com/mkempka/hyperbola-pde-build-demo.git
• Adjust values in org.eclipsercp.hyperbola.releng/custom.properties
• Execute org.eclipsercp.hyperbola.releng/build.xml with ant

Here are my slides from my talk at the Eclipse Testing Day 2011 that include some documentation. I’ll do more documentation in separate blog entries the next weeks.

I was astonished to find nothing in the SWTBot code or documentation that suggests how to automatically capture a screenshot when a SWTBot test fails. This is why I came up with a JUnit4 Rule that does this.

In your SWTBot test class include the field

public class MyTest {
  @Rule
  public CaptureScreenshotOnFailure screenshot = new CaptureScreenshotOnFailure();

  @Test public void mytestmethod() {
    SWTBot bot = new SWTBot(...);
    // swtbot clicks and types
  }

So, if now mytestmethod fails, a screenshot will be put in the current directory with the name mypackage.MyTest.mytestmethod.png

This is the rule, put it somewhere where your test classes can access it:

import org.eclipse.swtbot.eclipse.finder.SWTWorkbenchBot;
import org.junit.rules.MethodRule;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.Statement;

public class CaptureScreenshotOnFailure implements MethodRule {
	public final Statement apply(final Statement base,
			final FrameworkMethod method, Object target) {
		return new Statement() {
			@Override
			public void evaluate() throws Throwable {
				try {
					base.evaluate();
				} catch (Throwable onHold) {
					String fileName = constructFilename(method);
					new SWTWorkbenchBot().captureScreenshot(fileName);
					throw onHold;
				}
			}

			private String constructFilename(final FrameworkMethod method) {
				return "./"
						+ method.getMethod().getDeclaringClass()
								.getCanonicalName() + "."
						+ method.getName() + ".png";
			}
		};
	}
}

I think this makes writing tests with SWTBot more comfortable. Do you see ways to improve this rule?

Background

Good test suites

• are easy to set up
• pick up new tests as they are included in the projects (no need to include them manually)
• can differentiate between long running tests and fast unit tests
• Can be used in both the IDE and the continuous integration
• Don’t add performance

One way to create dynamic test suites is the ClassPathSuite. It goes through the classpath and adds all tests it can find to a test suite which is then executed. This approach has two problems that both come from loading all classes before inspecting them:

• The time to dynamically create the test suite goes up
• Static initializers may have wrong assumptions

For OSGi applications there is another way to create a test suite dynamically, thats the BundleTestCollector from Patrick Paulin. It is a similar approach in that it goes through specified bundles and adds classes to a test suite, but has neither of the above drawbacks

• No class is loaded before a test is actually executed
• There is no need to scan dependencies: You can narrow down the plug-ins that are scanned for tests

Sounds almost perfect, doesn’t it? Hold your breath, it has some problems, too, but they are small against everything else I stumbled across.

First, Patricks version only works with JUnit 3. I used a modified version for a while now that works with JUnit 4 and has some more assertions than Patricks original version. The remaining problem is, when looking at the test results in the JUnit view, the stack trace is not linked to the workspace. A double-click fails, and I need to open the class and navigate to the line by hand.

The BundleTestCollector requires OSGi Bundles. You can’t just run it as JUnit test. In your IDE it must be started as PDE Test (or SWTBotTest), and in the continuous integration, you should use the Eclipse Testing Framework or the SWTBot test runners.

The BundleTestCollector

Patrick published his version of the BundleTestCollector under the EPL, so my modifications are EPL, too. Here is a zip containing the BundleTestCollector and it’s single dependency, an Activator. You’ll have to copy them into an actual bundle and adjust the namespace. JUnit4 must be on the bundle dependency path.

The BundleTestCollector picks up the tests by naming conventions. If your test doesn’t adhere to a naming convention, it won’t be picked up.

Here is how to define a TestSuite. It’s still a JUnit 3 test suite, but the actual tests may be both, JUnit 3 or JUnit 4.

public class MyTestSuite extends TestSuite {

  public static Test suite() {
    BundleTestCollector testCollector = new BundleTestCollector();
    TestSuite suite = new TestSuite( "All Tests" );
    testCollector.collectTests(suite, "my.bundle.namespace", "my.package.namespace", "*Test");
    // add more lines collectTests(...) calls if necessary
    return suite;
  }
}

Hope that eases some of your test suite maintenance pains.

A lot of programmers seem to have difficulties with finding the right level for testing existing or new functionality. In general, they include too many classes and a too broadly defined system environment in test dependencies. This leads to trouble with reproducibility and stability.

In my talk, I’ll focus on RCP developers and share a few patterns and best practices that have been helpful in our team for finding the right level for writing a test.

Hope to see you in Darmstadt.

With some of my workspaces I want to have a similar IDE as with others, but some IDEs require special plug-ins. A while ago, as I still unzipped Eclipse-downloads, this was a huge pain. Every time I wanted to work in a specific workspace I had to remember which IDE I used for what, then find the workspace location on the disk, before I could do anything.

Permanent workspaces

Now I double-click a .yoxos file on my Desktop, then start working. Related .yoxos files hang out together on different areas on my desktop. This is possible with the Yoxos 5 Launcher which I explained in my last blog post.

I associate the workspace I want to start in terms of “upper right” or “vaguely in the middle, left” on the screen. No need to remember long directory names.
Remember, a .yoxos file is a definition of both the workspace and the IDE that works on the workspace. The actual workspace is somewhere in my home directory. Since every .yoxos file defines a separate IDE, I always click “Use this as default” after defining the first time where the workspace is located.

Temporary workspaces

Throwaway workspaces go to /tmp. Along with everything else in /tmp, they will be deleted the next time I reboot. The .yoxos file that defines a throwaway workspace should be deleted with it, so this belongs to /tmp as well. The IDE definition works this way:

• Start the Yoxos Launcher
• Add “Project SDK” and all the other desired plug-ins
• Save the .yoxos file to “/tmp/throwaway-workspace” (a new empty directory)
• Hit “Launch”

With the bundle pool I don’t worry about the plug-ins that compose the throwaway IDE. Only rarely something new gets downloaded anyway.

.yoxos files and workspaces

A special handling of .yoxos files in otherwise empty directories supports this workflow. If a .yoxos file is is started while residing in an empty directory, the IDE uses this directory as workspace. This provides an easy answer to the question about the “where”, and I use this feature at a regular basis.

On Mac OS X, I can append the extension “.yoxosws” to a directory that contains a .yoxos file. This defines a workspace that I can start directly with a double-click, without bothering about opening a folder to access the .yoxos file.

Conclusion

The Yoxos 5 Launcher makes it simple to handle a multitude of workspaces. The best thing is that you can stop wondering about the IDE contents and start thinking about workspaces. The Launcher provides a consistent UI to define new IDEs, including a huge number of 3rd-party plug-ins that are not shipped with the default Eclipse downloads.

But the number of useful plug-ins increased, and many are not included in the downloads from eclipse.org. Developers use different plug-ins in different workspace. For some developers, this leads to as many Eclipse installations as workspaces. Others capitulated and just don’t use many plug-ins even though they see their value; but managing the installations is just too hard. Others again have one huge installation that includes about everything for all the workspaces, and they too have pain with plug-in dependencies. They all suffer from plug-in dependencies.

Imagine you had a system where each plug-in you use is downloaded just once and reused whenever you need it for a new Eclipse IDE.

Yoxos 5 provides that.

Imagine you could just start your workspace and your IDE starts up including all plug-ins you want to work with in that workspace. If it is a new workspace you’d have automatically adjusted predefined settings, import projects etc.

Yoxos 5 excels at that.

Yoxos 5 unifies the workspace settings and its IDE description in a Yoxos Profile. A Yoxos Profile can be defined in a .yoxos file. The Yoxos Launcher creates Yoxos Profiles and starts them, for example when double-clicking the .yoxos file. Plug-ins are downloaded to the bundle pool and started only if the profile includes them.

Yoxos 5 is now in beta phase. You can try it out now:

• Download and install the Yoxos Launcher
• Download and start one of the sample profiles (further down at the download page)

Get more information at http://eclipsesource.com/yoxos5

1. My computer has 2 cores (there will be more in the future).
2. My IDE feels sluggier with every version.

Applications feel fast if the time between user action and application reaction is short. Ben Galbraith suggests that this threshold is around 200 ms for web applications, for desktop applications the boundary is lower. A rule of thumb places it around 100 ms, so I there must be a yellow range in between.

My IDE crosses this threshold more often than it should, so it’s sluggish. What’s the reason and what can be done about it?

It seems like a fundamental law in software engineering that every new version of any program executes more code than the previous version.

Java 5 introduced the java.util.concurrent API that made corse grained parallelism easier. The combination of thread pools, Futures and Barriers covered a wide range of tasks between CPU and I/O bound.

To me as UI programmer this had little consequence. For a while I could still state that Java+SWT is fast, consuming my part of that lunch that is not free any more but was cheap enough anyway in the doses I required.

This is changing. A few years ago we saw a pain barrier reached and solved in startup time of various operating systems. Today my pain barrier is reached in many of the functions that are executed as reaction to user clicks and key presses in my IDE. Undoubtedly it is only a matter of time when this will apply to smaller desktop applications, too. They must react, and using multiple cores is the solution at hand.

Other languages and frameworks already have good support for multicore programming, Java 7 will include the ForkJoin framework for Java. In the wild, it is a library known as jsr166y that can also be used with Java 6.

In the following I’ll discuss a little example that shows how to use the ForkJoin framework.

A simple easily parallelizable example is assigning random values in a large array. It’s also an example that can not be solved with raw processor power but accesses memory.

First, care must be taken where the random number comes from. Even a sequential solution using ForkJoins ThreadLocalRandom was about twice as fast than one with Math.random(). It’s also more comfortable to use.

    int[] result = new int[ARR_SIZE];
    for( int i = 0; i < result.length; i++ ) {
      result[i] = ThreadLocalRandom.current().nextInt( result.length );
    }

ForkJoin provides a class named ForkJoinPool as executor service for tasks. The constructor can be configured with the parallelism (number of threads) to be used. The default constructor takes the number of available processors as returned by Runtime.availableProcessors(). The pool is then given tasks for execution.

    ForkJoinPool pool = new ForkJoinPool();
    pool.invoke( new ArrayGenerator( result, 0, result.length ) );

The ArrayGenerator is a RecursiveAction. The RecursiveAction differs from the RecursiveTask mainly in the return value of the compute() method. It’s implementation is a straightforward recursive algorithm. First, there is an exit condition that solves the remaining problem sequentially. If that is not hit yet, the remaining problem is divided for two other RecursiveActions. Those are given to invokeAll, which blocks execution in this task until all subtasks are executed. In my experiments, the definition of the exit condition didn’t matter as long as all cores where busy at least once.

public class ArrayGenerator extends RecursiveAction {
 
  private int[] nums;
  private final int start, end;
 
  public ArrayGenerator( int[] nums, int start, int end ) {
    this.nums = nums;
    this.start = start;
    this.end = end;
  }
 
  @Override
  protected void compute() {
    if( arrayIsSmallEnough() ) {
      assignSequential();
    } else {
      int middle = start + (end - start)/2;
      ArrayGenerator left = new ArrayGenerator(nums, start, middle);
      ArrayGenerator right = new ArrayGenerator(nums, middle, end);
      invokeAll( left, right );
    }
  }
 ...
}

On my machine with 2 cores I achieved a speedup of 1.24 over the sequential solution. While that’s not as good as I hoped it still is a significant improvement that has the chance to move my clicks back down into the green area. I suspect that more cores could achieve a better speedup, so there is free lunch again once I design my program for concurrent execution.

Programming with this framework seems easy enough to me to see it adopted widely. However, we must rethink our problems and search for chances to parallelize.
I implemented the same thing with ThreadPoolExecutors and Futures. While the execution time did not differ there, it took me 3 times as long to implement it.

The concept of Import-Package is quite appealing. In today’s world, Java itself doesn’t have a concept for modules so we depend on OSGi for modularity. Actual dependencies on the Java side are specified on the class and package level. So it feels more natural to specify what you need via Import-Package instead of where you want something from via Require-Bundle.

As an experienced Eclipse developer dealing with many large projects… I’ve lost some enthusiasm for Import-Package. All those different bundles and their dependencies are difficult enough to grasp for newcomers. Require-Bundle has better tooling support in Eclipse and less complexity in general… so I recommend to use Require-Bundle unless someone needs the improved flexibility of Import-Package.

What’s the reason for my attitude?

Recently I spent some time with a single-sourcing a RAP/RCP project. Having different target platforms for the same bundles is a perfect example where you need the flexibility of Import-Package. Flexibility is good, right? So I went ahead and declared all dependencies as Import-Package.

Now imagine you have declared an Import-Package to the org.eclipse.ui bundle, being sure that org.eclipse.ui is exported in your target.

Everything is fine, right? So far so good in the typical Eclipse RCP case.

However, it’s not so fine if you try it with RAP although you’ll find that bundle org.eclipse.rap.ui.workbench exports the org.eclipse.ui package.

So the situation is:

my.bundle imports org.eclipse.ui
org.eclipse.rap.ui.workbench exports org.eclipse.ui

However, your bundle just won’t be resolved due to a missing Import-Package constraint on org.eclipse.ui.

The diagnostics in Equinox don’t help much either… it just tells you exactly the above imports and exports. I don’t know if there are better tools out there to diagnose issues.

The reason for this behavior is a concept in Java called split packages. It allows the same package to be declared across several bundles… giving each split a special name. So it’s something like a sub-package concept for packages. Although it is discouraged in the OSGi specification… it is used by the Eclipse workbench and other areas where there has been a lot of refactoring and Require-Bundle usage.

The correct import declaration in my.bundle would be

Import-Package: org.eclipse.ui; ui.workbench="split"

Why does it work in the RCP target case?

It wouldn’t if your target platform contains RCP only. However, most RCP applications have the bundle org.eclipse.ui.ide in their target platform which exports another split of the org.eclipse.ui package. OSGi only resolves split packages if at least two exported split packages can be found.

Knowing this… I now can safely use Import-Package for my single-sourced applications. However, I don’t see a way how I possibly could recommend this to students in my training courses or new people to OSGi in general. I’d need to tell them “Look, Require-Bundle is something that has been used in Eclipse for awhile, mostly for legacy reasons. We don’t recommend its use any more. Import-Package is better if you want looser coupling between bundles. However, be aware of the pain split packages can cause.”

I guess my gripe is that split packages make things complicated and we need better tooling to support them.

How are other people handling split packages in the wild?