Q7 Basic Tutorial

This tutorial gives an introduction to the UI testing tool Q7. Q7 is focused on testing Eclipse plugins and Eclipse RCP applications. The tutorial introduces all the basic steps to evaluate Q7 for your own project.

Why UI Testing?

The higher-level the test, the more difficult it usually is to automate it. In a good architecture, individual classes can be easily tested with unit tests. So too can the interaction of individual components such as a client-server communication. Unfortunately, it is more difficult to automate tests when the user interface is included. For this reason, system and acceptance tests can often only be carried out manually. This usually results in either high costs or a lack of test coverage or, more commonly, both. Because manual tests consistently create high costs per execution, regression testing cannot be carried out frequently. As a result, the interface with which the end user directly interacts is tested the least.

However, the user interface often creates the most visible value for the end-customers. What use is it to a clerk in an insurance company if a database query correctly calculates and returns results from the server but the table that displays the result is not updated? In his view, the feature is worthless, even if 99% of the functionality is working correctly.

The remedy for the problem above is applying UI testing tools. These simulate user tests of the complete system, including the UI. Use cases are automated and can be repeated frequently without the intervention of a real user. The Q7 tool enables the automation of UI tests for Eclipse plugins and Eclipse RCP applications.

Installation

To get started, you need to download a version of Q7. Exclusively for readers of this tutorial, an extended evaluation license of the full version is offered (see below). The free community edition allows users to record test cases but not to edit the script. In order to try out Q7, an application to be tested is needed. This can be any custom product. In this tutorial, we are using a demo application of the EMF Client Platform as an application under test. To understand all the steps of the article, this demo application can be downloaded from https://eclipse.org/emfclient/download.php (under demo products).

Q7 itself is an Eclipse plugin that re-uses much of the functionality of the Java IDE. Just as when you first start an IDE, a workspace needs to be configured. In this workspace, files such as tests or test suites, are stored as text files. Thus they can be versioned with any conventional SCM. For this, a corresponding client, for example EGit or subversion, can be installed directly in Q7. Furthermore, when you first start Q7, you must enter a license code. Click the following link to find an extended evaluation license exclusive for readers of this tutorial.

https://www.xored.com/products/q7/q7-promo-downloads/?code=RCRZC

In addition, there is a discount of 10% on the full version of Q7 exclusive for EclipseSource visitors available using the following coupon code: RCRZC (order under https://www.xored.com/products/q7/).

Application Under Test (AUT)

In the first step, the “Application Under Test” (AUT) is defined; this is the product being tested. The AUT should be available as a built version. To make it known to Q7, create an entry in the view “Applications”. The location parameter points to the folder containing the executable product. The name can be assigned at your discretion. The settings for an AUT can be adjusted later if necessary (right click => Configure). With the “Advanced” link, you can open the well-known dialogue from the Eclipse IDE to configure startup options where you can set options such as the JRE startup parameter or the workspace of the application.

Figure 1

After configuration, the AUT should be started once to see if it can be launched from Q7 correctly (right click => Run). This will take a little while the first time. Q7 generates a launch configuration that contains the product to be tested and some control plugins from Q7. In addition, the byte code of the application is so instrumented that Q7 can control the AUT automatically. Once the application has started, the configuration is finished and you can begin to record test cases.

The First Testcase

Test cases and all other described artifacts are managed in projects. Initially you create a new Q7 Project in the Q7 Explorer. The Q7 artifacts “TestCase”, “Test Suite” and “Context” are created with a right-click and, optionally, can be structured through “Folders”. Since all the used artifacts in Q7 are based on text files, a Q7 project can be versioned by using version management through the menu item “team”, just like in any Java project.

Figure 2

Once a test case has been created, you can assign a name, tags or a description. The field “Contexts” will be explained later. The field “Script” describes the actual test case. The recording of a test case is started via the “Record” button.

If the AUT is not running yet, Q7 will ask you which AUT to start. The Q7 control panel appears next to the running AUT. The control panel displays the results of the recording in real-time. Now you can perform the actual test in the AUT. This means that you can execute a use case as you would do in manual test. In parallel, the recording of the test case can be observed in the Control Panel. If you use the EMF Client Platform Demo application, you can create a new project, for example by right-clicking in the Model Explorer. The Control Panel will show the following recorded script:

After performing a use case step, a proper test must assess whether the User Interface displays the expected result. In the sample application, exactly one project called “Test Project” should now be visible in the Model Explorer view. Such checks can be performed easily in Q7 during the actual recording. To do so, activate the “Assertion Mode” (Alt + Shift +7).

The Assertion Mode enables you to select SWT UI items, such as a tree, a TreeItem or a label. In the dialogue boxes opened by Q7, the values of the selected SWT item can now be accessed. Figure 3 shows the selected TreeItem that has just been created manually. Now you can compare individual values of the SWT element with the expected values.

By default, the current value will be offered as an expected value. The example is validated by selecting the value of the attribute “caption” to check whether the TreeItem shows the text “Test Project”. By adding this assertion, you check not only the displayed value but also, indirectly, that there is a corresponding TreeItem. If you also want to validate that there is exactly one project, you can choose the whole Tree in Assertion Mode and check the value “ItemCount”. The Control Panel will show the following recorded lines:

Figure 3

Of course, you can also use other operators for comparison, such as “contains”, “does not equal” and so on. In many cases, however, the mere existence of a UI element is already sufficient as a test. The EMF Client Platform Demo application does not allow, for example, creation of two projects with the same name. This behavior can be checked with Q7 if a second project called “test project” is created. In this case, the project wizard will gray out the “Finish” button and display an error message. Both can be validated in the assertion mode with one click.

Figure 4

At the end of the simple example, the initial state of the application will now be restored, thus the test can be repeated. At the end of the test case, the deletion of the project is recorded and subsequently the tree is checked to see whether it is empty again:

Set-up and tear-down can, in principle, also be automated; more in the section Context. The recording is now complete in the control panel and the generated test case is stored. The test case can now be executed with a click on “Run” in Q7. The actual execution can be observed in the running AUT. The Execution View in Q7 will show the result, similar to the JUnit Test Result View. Q7 allows for managing multiple test cases, grouping them into tests suites, in a similar way to JUnit.

Test cases can be extended later by adding steps. When “Record” is clicked in an existing test case, the existing script will be executed. After this, additional steps can be recorded.

The Eclipse Command Language

Test cases in Q7 are recorded in the Eclipse Command Language (ECL), as observed in the control panel. ECL itself is independent of Q7 and is available as an Open Source project. ECL allows testers to adapt test cases after recording or even to write them manually.

While editing of ECL scripts manually, the user is supported by syntax-highlighting and auto-completion. In practice, however, most of the test cases can be recorded and then supplemented in some areas by the scripting. Because test cases in ECL are saved as text files, they can be versioned and maintained easily.

For example, to change the name of a view, users just need to replace the string Q7 uses to reference it, e.g., “get-view “Name of View””. By editing the ECL scripts, asserts that do not comply with the current state of the application can also be inserted. This is useful if a failure has been discovered and the assertion should not reference the wrong value. The test case will succeed again only after the bug has been corrected.

Additionally, ECL provides some commands, which cannot be recorded directly, such as loops and time measurements. In the following example, a test case is repeated 10 times. The example constrains how long the execution of the test case may take and thereby tests the performance of the application.

In this example, the part of the test case that is repeated can be recorded. The loop and the time measurement can be inserted using ECL. To debug more complicated test cases, break points can be used, as used in the Eclipse IDE. This allows the tester to track the execution of the test case step by step. ECL even allows you to define your own custom commands. Custom commands can be used to trigger actions, which are not visible in the user interface. Additionally, custom commands can be used to better support custom UI components.

Test Context

In addition to test cases and test suites, Q7 supports another artifact, the test context. A test context is the same as a setup method, as it is known from JUnit. Thus, certain conditions in the application, which are pre-requisite for a number of test cases, can be set-up. Q7 supports various types of contexts. For example, certain AUT workspaces or the Eclipse Preferences can be loaded and set accordingly. A context itself can also be an ECL script and as such be recorded.

The previously used example test case creates a project, so it can be used as a context for all other test cases that require an existing project. You may also want to define a context for the cleanup of the application, which includes, for example, closing editors and opening dialogue, or resetting the perspective. And application-specific cleanup can also be automated by ECL. For example, the following script deletes all projects in the demo application:

When clean-up operations are not executable in every state of the application, for example when logging out of a server, the corresponding operations should be executed in a try block. Thus Q7 will attempt to execute the corresponding part of the script but not indicate an error when individual parts are not executable. If one or more contexts are added to a test case, they are always executed before the test case is recorded. Thus, the actual recording of test cases starts directly in a defined state.

Q7 in Comparison

There are many UI testing tools, each of which has its advantages and disadvantages. In addition, the type of software being tested, the qualifications of the testers and the type of planned UI-tests play a role. The decision to use a certain tool for your project can only be made via an extensive evaluation. This tutorial can only highlight key features of Q7 compared to its competitors and the view of the authors. We do not claim to be comprehensive or objective.

A first field in which differences from other tools are obvious is the start of the Application Under Test (AUT). This must be instrumented by most UI testing tools, including Q7, to remotely control it. While in some competing products the instrumentation must be added or configured manually or additional programs need to be started, Q7 automates the instrumentation.

To control the application, Q7 instruments the byte code of the AUT. Thus Q7 receives each SWT event and can respond to these events and wait if necessary. As a consequence, Q7 is not dependent on how long a triggered UI operation actually takes. In other tools, it must be explicitly stated, how long to wait for a result.

Another striking difference between the Q7 and many other UI testing tools is the preferred method of creating test cases as well as their format. In other tools, test cases are often assembled from self-defined and reusable building blocks. These blocks are each mapped to actual UI elements. While recording is possible in other tools, it is not generally favored. Q7 is focused on recording test cases and, if necessary, the subsequent processing of ECL scripts. It does not provide an additional mapping between test cases and the actual UI. While reusable building blocks for the set-up can be created with test contexts, there is no real library of reusable test steps.

These very different paradigms have some consequences. Because recording in Q7 plays a central role, the recording in SWT applications works better than in almost any competing product. This approach allows the creation of numerous test cases in a relatively short time. These test cases are robust against layout changes but not against radical adaptations of the UI. In other tools, the predefined mapping can be adapted following changes in the UI, ideally without modifying any test case that uses the UI element. Furthermore, other tools, which work with abstract test models, offer some more options to parameterize test cases. Thus, for example, the same test can be repeated many times with different values. In Q7 something similar is possible in theory, but not without custom extensions.

The most effective approach depends very much on your application. If the goal is to achieve a high level of test coverage of a UI with many different elements, recording may be more efficient. If you need to test rather fewer UI elements with many different values and combinations, it seems reasonable to use a detailed mapping parameterization.

Finally, it is important that a UI test tool support the UI components used in your own application and is therefore able to test them. In particular, custom widgets and UI components from third-party vendors are critical. In principle, nearly all UI test tools can be extended to “understand” new UI elements, but this always creates extra effort, which must be weighed against the benefits. Due to the active development of Q7, coverage of supported widgets is quite high compared to some competitors. For example Nebula widgets or GEF usually work without modification. It is advisable, however, to run an evaluation that covers most “unusual” UI elements for each UI test tool under consideration.

Upshot

Q7 is an interesting alternative to the known test tools in the Eclipse environment. The tool stands out for its ease of use and a very effective recording of test cases. With Q7, test cases can be partially recorded in a similar time dimension in which they are manually tested once or twice. Q7 does not provide some advanced features that are offered by other testing tools, such as the parameterization of test cases or additional mapping of test steps on UI components. However, or even because of this, Q7 remains very easy to use. The recording of test cases is very similar to manually testing use cases in the AUT. Thus the use of Q7 requires very few computer science skills. The following adaptation of ECL Scripts, however, requires some experience.

Q7, minus the ability to edit ECL script free, is freely available as a community edition. The full version, as well as build server integration (Q7 runner), is pay-to-use. Furthermore, Q7 is offered for free for open source projects. Exclusively for visitors to EclipseSource, Xored offers an extended evaluation license as well as a discount on the full version.

Click the following link to find an extended evaluation license exclusive for readers of this tutorial.

https://www.xored.com/products/q7/q7-promo-downloads/?code=RCRZC

In addition, there is a discount of 10% on the full version of Q7 exclusive for EclipseSource visitors available using the following coupon code: RCRZC (order under https://www.xored.com/products/q7/).