The new Eclipse download wizard and RAP performance

The new Eclipse download wizard and RAP performance

The Eclipse Packaging Project is working on a wizard that allows you to build your own customizable download, combining plug-ins from predefined packages and Ganymede. The wizard uses Eclipse runtime technology – namely RAP for the web frontend and Equinox P2 for the installer. Before exposing the wizard to a wider audience, we had one burning question: Does it scale up to the volume of downloads being served by Eclipse.org? Read on to find out.

Load Test – Setup

We conducted a load test on the wizard and would like to share the results with you.

The tests were executed on a modern desktop computer with a Core2Duo 2.8 GHz CPU running Ubuntu Hardy. The JVM used a maximum of 512 MB of heap space and ran the concurrent marksweep garbage collector (-UseConcMarkSweepGC). The tests were run using JMeter 3.2 with 200 concurrent threads. Tomcat 6 was used for the server.

Load Test – Results

The results are promising: with a throughput of 100 requests per second we achieved an average response time of 14 miliseconds. Configuring a download takes up to 80 requests, which means we can serve approximately 100,000 downloads per day with a similar configuration.

Here’s a JMeter screenshot with more details on the results:

screen01

The fast average response time is paired with a low standard deviation of 60-80 milliseconds. The maximum response times were caused by full garbage collections, but only a few users should experience these response times (otherwise we would have higher average response times).

The allocation of heap is visualized in the diagram below, after multiple hours of load testing. A heap space of 250 MB was constantly needed for the approximately 1,000 concurrent user sessions that were simulated (250KB per session). Another 200 MB was built up and collected every one to two minutes.

screen02

Cloud Tests

To demonstrate performance we also ran tests against the same application running on the smallest available type of Amazon EC2 instance.
We ran the same load test as before – but using only 50 concurrent threads. After a period of 16 hours 16,000 sessions had been served, which means a throughput of about 24,000 sessions a day. This is almost equivalent to all the downloads served from Eclipse.org on a busy day. Average response time was approximately 100ms, with network latency accounting for most of the response time. CPU utilization on the server was around 15%.

Conclusion

We’re very positive about the results so far. Even though there is a memory overhead associated with RAP applications, in practice the performance is very solid. Random sampling of the application via a browser in parallel with the load tests showed that responsiveness was very good.

But don’t just take our word for it – try it out for yourself. The application is hosted on an EC2 instance reachable at
http://eppwizard.dyndns.org/eppwizard/rap.
The load test is still running with 50 concurrent sessions against that same instance.

We plan to beta test the wizard with the “Friends of Eclipse” after seeking initial feedback from the community on both usability and performance.
Take it for a spin and tell us what you think.

13 Comments
  • Anonymous
    Reply
    Posted at 9:47 am, September 17, 2008

    Is the source code somewhere available?

  • Scott Lewis
    Reply
    Posted at 4:55 pm, September 17, 2008

    Hi Manuel,

    I’ve been working on creating a minimal Equinox P2 configuration that runs as a war (in tomcat, etc) using the servletbridge. The minimal configuration supports p2-based installation (via osgi console).

    If possible, I would like to test this out by installing this rap-based installer. This is pretty easily done, if your installer app can be exposed as a feature or update site.

    Would this be possible? Please contact me at slewis at composent.com or slewis at code9.com.

    Thanks!

  • SeuAndré
    Reply
    Posted at 11:05 pm, September 17, 2008

    very nice!!! 🙂

  • Manuel Woelker
    Reply
    Posted at 7:00 am, September 18, 2008

    @anonymous
    The source code is in the eclipse CVS repositories at :pserver:anonymous@dev.eclipse.org:/cvsroot/technology in the org.eclipse.epp package.

    To get the demonstration running take a look at
    http://wiki.eclipse.org/EPP/Wizard/Demonstration_HowTo which has basic documentation on how to get started, as well as a team project set with all the necessary components.

    As this is still work in progress sources, wiki and the running demonstration may not be quite in sync.

    Hopefully I’ll be able to update everything by the end of the week.

  • oisin
    Reply
    Posted at 12:00 pm, September 18, 2008

    This is pretty cool, but in my couple of trial runs the downloads have frozen about three-quarters the way through 🙁

    That being said, I’m still eager to follow what’s going on 🙂

  • Manuel Woelker
    Reply
    Posted at 3:50 pm, September 18, 2008

    @oisin:
    I am assuming you are referring to the actual installation with the p2 Installer, not the download of the 5MB installer itself.

    I have experienced some problems as well, and we’ll look into these issues. Do you know if it was a specific combination of components you were selecting? The logs created by the p2 installer might also be helpful for identify the problem.

    My primary concern right now is with the wizard, but once that is done, we will need to adress the issues concerning the p2 installation process, like scalability, availability of mirros, etc.
    Sometimes the download is just painfully slow and takes almost literally ages. This will need to improve to make this wizard based installation a viable alternative to the traditional downloads.

  • oisin
    Reply
    Posted at 4:31 pm, September 18, 2008

    Yes, I am referring to the actual installation, which was moderately large. If I can find the p2 log, I can check it out, or send it on. Where should I look for it?

    It may be that the problem is a network congestion issue, and I just haven’t had the time to check into it further.

  • Manuel Woelker
    Reply
    Posted at 7:22 am, September 19, 2008

    @oisin:
    The .log file should be in the configuration subfolder of the download p2 installer.
    Other things that would be of help are the installer.properties file in the downloaded archive, as well as which platform you are running on.
    I can be contacted at mwoelkerXinnoopractXcom.
    Thanks for your effort!

  • oisin
    Reply
    Posted at 9:23 am, September 19, 2008

    For those of you watching, there was no relevant errors or failures in the log, so it looks like a network transient rather than anything buggy on the RAP or p2 side of the affair 🙂

  • stefan.hansel
    Reply
    Posted at 1:48 pm, October 5, 2008

    Manuel,

    it seems, that you have put up quite a good infrastructure for load-testing and measuring results.
    Thanks for sharing the results.

    There is one open enhancement request regarding memory consumption (https://bugs.eclipse.org/bugs/show_bug.cgi?id=240143).

    The patches there have not yet been applied, because the RAP team is not sure, whether these could have negative impact on performance.

    Is there any chance, that these patches could be applied for a short while, just to have some arguments pro/against this patch ?

    With the analysis done so far by you, this is a great base for comparisons!

  • Manuel Woelker
    Reply
    Posted at 7:24 am, October 6, 2008

    Hi Stefan,

    thanks for your commment. Heap consumption (and heap turnaround) was indeed a problem for this use case. So this patch might help a lot in that regard. Unfortunately I won’t have the time to perform more tests in the near future, so I can’t promise if and when we can perform any benchmarking. Maybe the benchmarks would also have to be performed slightly different, since my main concern was overall throughput and responsiveness.

    For now the wizard seems to run “good enough” on stock RAP. We’ll keep that patch in mind though and maybe we can integrate it in the future either directly or via upstream updates in RAP. So thanks again for your post.

  • stefan.hansel
    Reply
    Posted at 5:13 pm, October 6, 2008

    What a pity … if you do load-testing again (or have memory-problems) in the future maybe you can think of me 😉 … I could also contribute a patched RAP-version for any version that is under test then, if that’s of any help.

    I’m afraid that the patch won’t be officially integrated as long as there is noone who can prove, that the throughput and responsiveness is not harmed.
    That’s why I thought your setting and the measures are perfect as they are.

    Anyway … a little bit of hope remains that somewhen in the unknown future there will be new tests 😀

  • Posted at 1:30 am, July 9, 2010

Post a Comment

Comment
Name
Email
Website