I was certainly looking forward to those days where each and every error log entry with a stack trace containing the text “cdo” somewhere automagically ends up in my bugzilla inbox!

And the users: They’d never again have to write all these boring addon infos like “What did I do?”, “How can you reproduce?” or “I looked at the code and here is my solution!”.

Why not become even better? What about an AutoFixMeAgent?

What I’d love to see is a) some better support inside eclipse for this – a base error reporter along w/ an extension point. Even better – would be a community (eclipse or some other place) project for a basic server-side report receiving, storage, and processing system – it should have some mechanisms for filtering out garbage, and an extension mechanism for customization (I could see it being used by the various RCP people and some eclipse downstream builds as the basis for in house servers).

1. What’s a failure? When a product like FireFox crashes, it just dies. One of the great things about Eclipse is that it rarely actually dies. Different parts and actions fail, but the product still keeps running. So how do you detect a failure event to report? What we did in Workshop is look for stack traces emitted to the log. When we detected an error log event with a stack trace, we would open a dialog informing the user of the failure and giving them a chance to send a report. The problem is that all failures are not equal in severity and some failures can come at a quite high clip with relatively little impact on what user is currently doing. In the first release of this system, we were fielding a lot of newsgroup posts from irate users who go sick of this dialog popping up in their face all the time. Some measures can be taken to mitigate this by detecting failures already reported or having rules about what can be ignored or letting users tell you to not bother them again. In the end all of these are just stop-gap measures. Eclipse environment requires a different approach. Perhaps you prompt the user to opt-in / opt-out on first failure and then proceed silently after that. You loose the ability for the user to type in comments about what they were doing, but we did not find that users were taking advantage of that feature anyway.

2. Signal to noise ratio. A good problem report needs sufficient information so that the dev studying it can reproduce it or at least make an educated guess as to the cause of the problem. It is extremely difficult for automated systems like this to produce good problem reports. You have the stack trace, but that only takes you so far. Gathering more information doesn’t necessarily help either. We could capture plugin listings, full stack dumps, etc., but unless we have some kind of a crazy simulator, it is unlikely that the developer will be able to make sense of the problem report. Expect only a tiny fraction of reports to be actionable. We were seeing something like 5% to 10%.

3. Separate system from bug database. You would not believe how quickly a system like this can accumulate problem reports. Even if you have a really good algorithm that can detect when two stack traces represent the same problem (not an easy thing to write without producing high rates of false positives), you should still expect to gather problems faster than your resources allow you to analyze them. There are a couple of lessons that fell out of this for us. (a) The action of turning an automated report into a bug report is something that human does during the problem triage. (b) The system needs to be able to prioritize reports somehow. A good rule is to count how many time a certain problem is reported. (c) The system needs to be able to shed old reports automatically. You really don’t care about problems reported two months ago. You have enough coming in this week to worry about.

4. Whose problem is it? This is actually an extremely difficult question to even approximate the answer to. Is stuff closer to the top of the stack trace more at fault than the stuff lower down? Not really. Not in all cases. An approach that worked decently is to define pattern tags that would match some portion of package name to whatever level of granularity we cared to apply in a particular case. The system would tag reports automatically on arrival with however many tags applied. The individual devs would setup queries looking for certain tags. For every problem report they could unset the tag if they don’t believe the problem is in their component. If the dev looking at the problem believes there is enough to go on, we had a button that automatically opened a case in the bug database and marked the problem report as handled. An approach like this could work well for Eclipse as it scales across projects and companies. This would require a single central problem report database maintained at eclipse.org that anyone could query.

5. Which release is it? Is it a dev build? Was there any unreleased / patched code running that may have contributed to the problem? We did not find a good answer to all aspects of this problem. It is one of the contributing factors that creates the low signal to noise ratio.

If there is interest creating such a system at Eclipse, I would be interested in contributing to that effort, but the system needs to be built in such a way that is inherently useful to both Eclipse projects and third parties extending Eclipse.