Introducing the Eclipse Theia Community Test Suite

In this article, we introduce the Eclipse Theia Community Test Suite. It provides adopters of Theia the ability to periodically execute  end to end tests on the latest versions of Theia. By adding tests, adopters are enabled to detect regressions or significant changes of Theia that are relevant to their Theia-based products as early as possible in the process of adopting new Theia versions. All adopters are invited to contribute to it and benefit from this joint effort!

Automated testing is a crucial ingredient for continuous quality assurance of any software product. In this context, finding the right balance between test coverage, effort of test development and maintenance is key. In general, we can distinguish between three levels of testing:

• Unit tests: Testing software components in isolation (e.g. classes)
• Integration tests: Testing the integration of multiple different software components in combination
• End-to-end tests: Testing the full software product as it would be deployed to the end user

All three types of tests have their particular value, typically the coverage should be highest for unit tests and integration tests, which are relatively easy to develop and maintain, usually cheap to execute, and due to their limited scope rather robust over time.

End-to-end tests, however, are usually the most tricky ones. On the plus site, they test the full software product as it is shipped to users and directly verify the software behavior on the user level, which arguably is the most important level. On the downside, end-to-end tests are more expensive to run, harder to debug and often less robust, thus more effort to maintain. Therefore, end-to-end tests are usually focused on the most important user flows.

Theia itself is a platform, and per se not a product. Thus, its testing strategy puts the main focus on unit and integration. After all, it is eventually the products that are built on top of Theia, which define the most important user flows. As a consequence, many adopters are creating and maintaining end-to-end test suites themselves for their own custom products based on Theia, which makes perfect sense.

Why do we need a Theia Community Test Suite for end-to-end tests then?

A Theia-based product is usually a combination of both, product-specific components and generic tool functionality that is provided by Theia off-the-shelf, such as the file explorer, the code editor or the toolbar. When testing a custom Theia-based product, you consequently will also need two types of tests. The first one tests custom features that are specific to your tool and might not be even open source. These tests need to be maintained and executed by the adopter. The second type of tests, however, is verifying generic behaviors and functionalities of Theia, which the custom product relies on. 

For this second group of tests – tests that validate Theia’s generic tool functionality –, there is a high overlap between adopting projects and this part of the end-to-end test suite could actually be shared. And this is precisely why we created the Theia Community Test Suite for end-to-end tests. The test suite is hosted as part of the Theia project and will be executed against master, so any failures will be detected very early. These test results will be a crucial input for guiding adopters on which of their user flows that involve generic functionality might be impacted, which versions are safe to upgrade to without adaptation, etc.

As only adopters can judge which parts of the system they want to cover, we open the test suite for all adopters and want to keep the barrier for contributions very low. Even if you have very specific things to validate, they can be added. In turn, the Theia core team will not take the maintenance for the test suite, and it will be a community effort to fix failing tests and to report bugs if tests discover regressions. Writing and maintaining tests should be rather straightforward as the test suite is based on Playwright and Theia’s explicit support for Playwright to simplify the development of robust tests.

We hope that the community test suite allows our adopters to better share their testing efforts, reduce their costs and of course increase the stability in the process of adopting new Theia versions. We are looking forward to feedback, contributions and any additional thoughts.

Last but not least, here it is, the Eclipse Theia Community Test Suite for End-To-End Tests

If you are interested in building custom tools or IDEs based on Eclipse Theia, EclipseSource provides consulting and implementation services for Eclipse Theia as well as for web-based tools in general. Furthermore, if you want to extend Theia with features, fixes or new tests, EclipseSource provides sponsored development for Theia, too. Get in contact with us, to discuss your use case!