RCP Testing Tool (RCPTT)

This tutorial gives an introduction to the RCP Testing Tool. The tool used to be commercial and was named “Q7”, in 2014 it has been renamed and released as an open source project. The RCP Testing Tool (RCPTT) is focused on testing Eclipse plugins and Eclipse RCP applications. The tutorial introduces all the basic steps to evaluate RCPTT for your own project.

Why UI Testing?

The higher-level the test, the more difficult it usually is to automate it. In a good architecture, individual classes can be easily tested with unit tests. So too can the interaction of individual components such as a client-server communication. Even UI components can and should be tested with small and isolated unit tests. Unfortunately, it is more difficult to automate tests when the system and its user interface are integrated. For this reason, system and acceptance tests can often only be carried out manually. This usually results in either high costs or a lack of test coverage or, more commonly, both. Because manual tests consistently create high costs per execution, regression testing cannot be carried out frequently. As a result, the interface with which the end user directly interacts is tested the least.

However, the user interface often creates the most visible value for the end-customers. What use is it to a clerk in an insurance company if a database query correctly calculates and returns results from the server but the table that displays the result is not updated? In his view, the feature is worthless, even if 99% of the functionality is working correctly.

The remedy for the problem above is applying UI testing tools. These simulate user tests of the complete system, including the UI. Use cases are automated and can be repeated frequently without the intervention of a real user. The RCPTT enables the automation of UI tests for Eclipse plugins and Eclipse RCP applications.

Installation

To get started, you need to download the latest version of RCPTT from the projects download page. In order to try out RCPTT, an application to be tested is needed. This can be any custom product. In this tutorial, we are using the Eclipse IDE as an application under test. This does not mean that RCPTT is restricted to create tests for the Eclipse IDE, any RCP application and be the application undert test. In fact, the Eclipse IDE is nothing but a RCP application itself.

To understand all the steps of the article, download an Eclipse IDE from the Eclipse download page. We are using the Eclipse Modeling Tools package, but any Java package should work as well.

RCPTT itself is an Eclipse plugin that re-uses much of the functionality of the Java IDE. Just as when you first start an IDE, a workspace needs to be configured. In this workspace, files such as tests or test suites, are stored as text files. Thus they can be versioned with any conventional SCM. For this, a corresponding client, for example EGit or subversion, can be installed directly in RCPTT.

Application Under Test (AUT)

In the first step, the “Application Under Test” (AUT) is defined; this is the product being tested. The AUT should be available as a built version. To make it known to RCPTT, create an entry in the view “Applications”. The location parameter points to the folder containing the executable product. The name can be assigned at your discretion. The settings for an AUT can be adjusted later if necessary (right click => Configure). With the “Advanced” link, you can open the well-known dialogue from the Eclipse IDE to configure startup options where you can set options such as the JRE startup parameter or the workspace of the application. In this tutorial, as we use the Eclipse IDE as an AUT, the configuration points to the directory, which contains the eclipse.exe file. If you test your own RCP application, point to the directory containing your product accordingly.

After configuration, the AUT should be started once to see if it can be launched from RCPTT correctly (right click => Run). This will take a little while the first time. RCPTT generates a launch configuration that contains the product to be tested and some control plugins from RCPTT . In addition, the byte code of the application is so instrumented that RCPTT can control the AUT automatically. In this tutorial, we use an Eclipse IDE as an AUT. When it starts the first time, a workspace location is requested. To enable a simple start-up for RCPTT, select a workspace and also check “Use this workspace as a default” so that with the next start-up the workspace choice is not necessary anymore. When the AUT has started, close the Welcome Screen. These preparation steps could also be automated with RCPTT using “contexts”, which we will introduce later in this tutorial. Once the application has started, the configuration is finished and you can begin to record test cases.

Want to learn more about Testing?

⇒ Have a look at our Eclipse Testing Training

The First Testcase

Test cases and all other described artifacts are managed in projects. Initially you create a new RCPTT Project in the RCPTT Explorer. The RCPTT artifacts “TestCase”, “Test Suite” and “Context” are created with a right-click and, optionally, can be structured through “Folders”. Since all the used artifacts in RCPTT are based on text files, a RCPTT project can be versioned by using version management through the menu item “team”, just like in any Java project.

Once a test case has been created, you can assign a name, tags or a description. The field “Contexts” will be explained later. The field “Script” describes the actual test case. The recording of a test case is started via the “Record” button (Top Right in the test case editor).

If the AUT is not running yet, RCPTT will ask you which AUT to start. The RCPTT control panel appears next to the running AUT. The control panel displays the results of the recording in real-time. Now you can perform the actual test in the AUT. This means that you can execute a use case as you would do in manual test. In parallel, the recording of the test case can be observed in the Control Panel. In our tutorial, as a first test case, we will create a new Java project named “testJavaProject” by right clicking in the Package Explorer. The Control Panel will show the following recorded script:

get-view "Package Explorer" | get-tree | get-menu "New/Java Project" | click
with [get-window "New Java Project"] {
   get-editbox -after [get-label "Project name:"] | set-text testJavaProject
   get-button Finish | click
}

After performing a use case step, a proper test must assess whether the User Interface displays the expected result. In the sample test case, exactly one project called “testJavaProject” should now be visible in the Package Explorer view. Such checks can be performed easily in RCPTT during the actual recording. To do so, activate the “Assertion Mode” (Alt + Shift +7).

The Assertion Mode enables you to select SWT UI items, such as text fields, a tree, a TreeItem or a label. In the dialogue boxes opened by RCPTT, the values of the selected SWT item can now be accessed. The following screenshot 3 shows the selected TreeItem that has just been created manually. Now you can compare individual values of the SWT element with the expected values.

By default, the current value will be offered as an expected value. The example is validated by selecting the value of the attribute “caption” to check whether the TreeItem shows the text “testJavaProject”. By adding this assertion, you check not only the displayed value but also, indirectly, that there is a corresponding TreeItem. If you also want to validate that there is exactly one project, you can choose the whole Tree in Assertion Mode and check the value “ItemCount”. The Control Panel will show the following recorded lines:

get-view "Package Explorer" | get-tree | get-item testJavaProject | get-property caption | equals testJavaProject | verify-true
get-view "Package Explorer" | get-tree | get-property itemCount | equals 1 | verify-true

Of course, you can also use other operators for comparison, such as “contains”, “does not equal” and so on. In many cases, however, the mere existence of a UI element is already sufficient as a test. The Eclipse IDE does not allow, for example, creation of two Java projects with the same name. This behavior can be checked with RCPTT if a second project called “testJavaProject” is created. In this case, the project wizard will gray out the “Finish” button and display an error message. Both can be validated in the assertion mode with one click.

get-view "Package Explorer" | get-tree | select testJavaProject | get-menu "New/Java Project" | click
get-window "New Java Project" | get-editbox -after [get-label "Project name:"] | set-text testJavaProject
get-window "New Java Project" | get-editbox -after [get-label "Create a Java Project"] | get-property text | equals "A project with this name already exists." | verify-true
with [get-window "New Java Project" | get-button Finish] {
get-property enablement | equals false | verify-true
get-property caption | equals "&Finish" | verify-true
}

At the end of this simple example, the initial state of the application will now be restored, thus the test can be repeated. At the end of the test case, the deletion of the project is recorded and subsequently the tree is checked to see whether it is empty again:

get-window "New Java Project" | get-button Cancel | click
get-view "Package Explorer" | get-tree | select testJavaProject | get-menu Delete | click
with [get-window "Delete Resources"] {
   get-button "Delete project contents on disk (cannot be undone)" | check
   get-button OK | click
}
get-view "Package Explorer" | get-tree | get-property itemCount | equals 0 | verify-true

Please note: Set-up and tear-down can and should also be automated, it should not be part of the test case in a real usage scenario; more in the section Context. The deletion of the project in this example is just for keeping things simple for now. The recording is now complete in the control panel and the generated test case is stored. The test case can now be executed with a click on “Run” in RCPTT. The actual execution can be observed in the running AUT. The Execution View in RCPTT will show the result, similar to the JUnit Test Result View. RCPTT allows for managing multiple test cases, grouping them into tests suites, in a similar way to JUnit.

Test cases can be extended later by adding steps. When “Record” is clicked in an existing test case, the existing script will be executed. After this, additional steps can be recorded.

Want to learn more about Testing?

⇒ Have a look at our Eclipse Testing Training

The Eclipse Command Language

Test cases in RCPTT are recorded in the Eclipse Command Language (ECL), as observed in the control panel. ECL itself is independent of RCPTT and is is also open source. ECL allows testers to adapt test cases after recording or even to write them manually.

While editing of ECL scripts manually, the user is supported by syntax-highlighting and auto-completion. In practice, however, most of the test cases can be recorded and then supplemented in some areas by the scripting. Because test cases in ECL are saved as text files, they can be versioned and maintained easily.

For example, to change the name of a view, users just need to replace the string ECL uses to reference it, e.g., “get-view “Name of View””. By editing the ECL scripts, asserts that do not comply with the current state of the application can also be inserted. This is useful if a failure has been discovered and the assertion should not reference the wrong value. The test case will succeed again only after the bug has been corrected.

Additionally, ECL provides some commands, which cannot be recorded directly, such as loops and time measurements. In the following example, a test case is repeated 10 times. The example constrains how long the execution of the test case may take and thereby tests the performance of the application.

measure-time "My Time Constraint"{
 repeat -times 10 -command{
 //Testscript
 }
 } | constrain-max duration 20000

In this example, the part of the test case that is repeated can be recorded. The loop and the time measurement can be inserted using ECL. To debug more complicated test cases, break points can be used, as used in the Eclipse IDE. This allows the tester to track the execution of the test case step by step. ECL even allows you to define your own custom commands. Custom commands can be used to trigger actions, which are not visible in the user interface. Additionally, custom commands can be used to better support custom UI components.

Test Context

In addition to test cases and test suites, RCPTT supports another artifact, the test context. A test context is the same as a setup method, as it is known from JUnit. Thus, certain conditions in the application, which are prerequisite for a number of test cases, can be set-up. RCPTT supports various types of contexts. For example, certain AUT workspaces or the Eclipse Preferences can be loaded and set accordingly. A context itself can also be an ECL script and as such be recorded.

The previously used example test case creates a project, so it can be used as a context for all other test cases that require an existing project. You may also want to define a context for the cleanup of the application, which includes, for example, closing editors and opening dialogue, or resetting the perspective. In our simple example, the context should empty the workspace and close all open editors. These two set-up steps are supported by the context type “workspace”.

Application-specific cleanups could also be automated by ECL. That means you can record or write ECL scripts, which execute the necessary operations, just like we did before to create the test case. When clean-up operations are not executable in every state of the application, for example when logging out of a server, the corresponding operations should be executed in a try block. Thus Q7 will attempt to execute the corresponding part of the script but not indicate an error when individual parts are not executable. If one or more contexts are added to a test case, they are always executed before the test case is recorded. Thus, the actual recording of test cases starts directly in a defined state.

Want to learn more about Testing?

⇒ Have a look at our Eclipse Testing Training

Procedures

Similar to functions or methods in other languages, ECL provides a means of code reuse through a construct called procedures. A procedure is a named block of ECL code which can receive an arbitrary number of arguments, and works with both named and ordered argument types.

A simple example is the following “hello world” procedure:

proc “hello-world” {
  show-alert “Hello, world!”
 }

To call it, all you have to do is simply use its defined name:

// shows “Hello, world!”
 hello-world

Arguments are declared in square brackets and it is possible to also assign them default values in their declaration.

proc “press-button” [val window] [val button “OK”] {
  get-window $window | get-button $button | click
 }
 
 // clicks “OK” on “New Project”
 press-button “New Project”
 
 // clicks “Cancel” on “New Project” (ordered and named invocation)
 press-button “New Project” “Cancel”
 press-button -window “New Project” -button “Cancel”

Procedures not only allow and encourage code reuse, modularization and enhance the readability and structure of ECL code, but also allow for setting up parameterized tests. Consider the following example:

proc “calculate-sum” [val a] [val b] {
   with [get-window “Calculator”] {
     get-editbox -after [get-label “A”] | set-text $a
     get-editbox -after [get-label “B”] | set-text $b
     get-button “OK” | click
   }
 }
 proc “verify-result” [val result] {
   with [get-window “Calculator”] {
     get-editbox -after [get-label “Result”] | get-property text | contains $result | verify-true
   }
 }
 
 calculate-sum 3 4
 verify-result 7
 
 calculate-sum 1 5
 verify-result 6

Procedures allow code to be reused not only within a particular test case, but also across multiple tests. This can be achieved by storing the procedures in contexts. Thus, they can be made available for a large number of tests, which can then take advantage of the shared code by referencing the contexts containing the needed procedures. As direct advantage of this , fixing problems or adding features in shared procedures only has to happen in one place, hugely improving maintainability.

RCPTT in Comparison

There are many UI testing tools, each of which has its advantages and disadvantages. In addition, the type of software being tested, the qualifications of the testers and the type of planned UI-tests play a role. The decision to use a certain tool for your project can only be made via an extensive evaluation. This tutorial can only highlight key features of RCPTT compared to its competitors and the view of the authors. We do not claim to be comprehensive or objective.

A first field in which differences from other tools are obvious is the start of the Application Under Test (AUT). This must be instrumented by most UI testing tools, including RCPTT, to remotely control it. While in some competing products the instrumentation must be added or configured manually or additional programs need to be started, RCPTT automates the instrumentation.

To control the application, RCPTT instruments the byte code of the AUT. Thus RCPTT receives each SWT event and can respond to these events and wait if necessary. As a consequence, RCPTT is not dependent on how long a triggered UI operation actually takes. In other tools, it must be explicitly stated, how long to wait for a result.

Another striking difference between the RCPTT and many other UI testing tools is the preferred method of creating test cases as well as their format. In other tools, test cases are often assembled from self-defined and reusable building blocks. These blocks are each mapped to actual UI elements. While recording is possible in other tools, it is not generally favored. RCPTT is focused on recording test cases and, if necessary, the subsequent processing of ECL scripts. It does not provide an additional mapping between test cases and the actual UI. While reusable building blocks for the set-up can be created with test contexts, there is no real library of reusable test steps.

These very different paradigms have some consequences. Because recording in RCPTT plays a central role, the recording in SWT applications works better than in almost any competing product. This approach allows the creation of numerous test cases in a relatively short time. These test cases are robust against layout changes but not against radical adaptations of the UI. In other tools, the predefined mapping can be adapted following changes in the UI, ideally without modifying any test case that uses the UI element. Furthermore, other tools, which work with abstract test models, offer some more options to parameterize test cases. Thus, for example, the same test can be repeated many times with different values. In RCPTT something similar is possible in theory, but not without extensive ECL knowledge and sometimes custom extensions.

The most effective approach depends very much on your application. If the goal is to achieve a high level of test coverage of a UI with many different elements, recording may be more efficient. If you need to test rather fewer UI elements with many different values and combinations, it seems reasonable to use a detailed mapping parameterization.

Finally, it is important that a UI test tool support the UI components used in your own application and is therefore able to test them. In particular, custom widgets and UI components from third-party vendors are critical. In principle, nearly all UI test tools can be extended to “understand” new UI elements, but this always creates extra effort, which must be weighed against the benefits. Due to the active development of RCPTT, coverage of supported widgets is quite high compared to some competitors. For example Nebula widgets or GEF usually work without modification. It is advisable, however, to run an evaluation that covers most “unusual” UI elements for each UI test tool under consideration.

Upshot

RCPTT is an interesting alternative to the known test tools in the Eclipse environment. The tool stands out for its ease of use and a very effective recording of test cases. With RCPTT, test cases can be partially recorded in a similar time dimension in which they are manually tested once or twice. RCPTT does not provide some advanced features that are offered by other testing tools, such as the parameterization of test cases or additional mapping of test steps on UI components. However, or even because of this, RCPTT remains very easy to use. The recording of test cases is very similar to manually testing use cases in the AUT. Thus the use of RCPTT requires very few computer science skills. The following adaptation of ECL Scripts, however, requires some experience.

RCPTT is freely available as a open source project. The build server integration (RCPTT CI Integration), is pay-to-use.