EMF Fuzz Testing

Many test cases for an application involve data defined by the data model. To achieve good test coverage it is customary to run the test cases for a variety of input data sets. This can be achieved by manually specifying input data. However, this is a complex task and it is easy to miss important input data and thereby miss bugs. Fuzz testing is about pseudo-randomly generated input data. A test case is written without specifying the exact input data, but rather with parameters driving the pseudo-random generation of input data for the test case. The test case can then be run a specified number of times with different seeds for the input data generation, thereby discovering bugs with input data you may not have selected manually. This post shows a fuzz testing framework for EMF models in action. It is based on JUnit 4 and existing tests can easily be migrated to run with it. We have applied this approach on many projects in practice including EMFStore and EMF Compare and have found the most astonishing bugs.

ModelMutator

First of all, this framework needs the ability to generate data based on a data model. We implemented a generator to create and mutate instances of EMF models in a deterministic way. The ModelMutator can generate instances of models based on a seed and different parameters. It can be used, for example, for performance tests with big models containing 1,000,000 objects:

ESModelMutatorConfiguration config = new ESModelMutatorConfiguration(ePackage, rootEObject, 1);
config.setMinObjectsCount(1000000);
ESDefaultModelMutator.generateModel(config);

The code fills the root EObject (root of the containment tree of the model instance) with 1,000,000 instances of EClasses declared in the given EPackage. It sets attribute values and cross-references based on the given seed to generate a well-connected graph with many different values.

Fuzz Testing Framework

To use the fuzz approach three ingredients are needed:

1. A JUnit TestRunner to repeatedly run the same test with different data (seeds).
2. A DataProvider to ‘inject’ different model instances into the test case.
3. The ModelMutator to provide different model instances to the DataProvider.

The basic fuzz JUnit runner executes a test repeatedly with different data. It is bound to a test case with the JUnit @RunWith annotation. The EMFDataProvider is driven by a set of annotations and injects data into a field annotated with @Data. The DataProvider itself is also registered with an annotation on the test case (@DataProvider annotation). The following snippet shows an example test using fuzz testing:

@RunWith(ESFuzzyRunner.class)
@DataProvider(ESEMFDataProvider.class)
public class FuzzyTest {
  @Data
  private EObject root;
  @Test
  public void test(){
    Assert.assertNotNull(root);
  }
}

Before the test() method is executed, the root object (@Data annotation) is already injected with generated data by the EMFDataProvider, so it will not be null. To configure parameters like the number of test runs there is a configuration for each test class which is stored in a file to ensure reproducibility:

This file can also be edited with a small EMF-based Editor (see screenshot).

After this quick configuration step, the test can be run with different pseudo-randomly generated data (see below).

As you can see, the test was executed 5,000 times in about 10 seconds. The fuzzy runner can be executed with any implementation of the interface FuzzyDataProvider - like the EMFDataProvider - and therefore can also be used for any kind of test with data independent of EMF. It is already in active use to test the EMFStore model repository and helped us to find and fix quite a few bugs. To know how good your test-cases with the generated data really are, it is always a good idea to measure test coverage, e.g. with EclEmma.

Getting started

To get started, checkout the source code from our git repositories at eclipse.org, which you’ll find on the Getting involved page of EMFStore. The plugins you need are available in two git repositories.

Plugins from org.eclipse.emf.emfstore.core.git repository:

• org.eclipse.emf.emfstore.modelmutator

Plugins from org.eclipse.emf.emfstore.other.git repository:

• org.eclipse.emf.emfstore.fuzzy
• org.eclipse.emf.emfstore.fuzzy.emf
• org.eclipse.emf.emfstore.fuzzy.emf.edit
• org.eclipse.emf.emfstore.fuzzy.emf.editor
• org.eclipse.emf.emfstore.fuzzy.emf.example: small example plugin configured to run with the example plugin, just run the FuzzyTest as JUnit Plug-in Test
• org.eclipse.emf.emfstore.fuzzy.emf.test: shows more advanced examples but also has a dependency to org.eclipse.emf.emfstore.client from the org.eclipse.emf.emfstore.core.git repository

We are currently working on making this available as an installable feature from a p2 update site, so stay tuned.

Happy Testing!