Eclipse Feedback Agent?

Eclipse Feedback Agent?

For those who follow me on Twitter, know that I and Firefox don’t get along. I just want to browse the web, and Firefox just wants to eat memory and crash all the time. However, I was pleasantly surprised when Firefox crashed and offered me a chance to give feedback:

Firefox Crash Reporter

I was so thrilled at this opportunity and thought about doing something like this in Eclipse land.

Does it make sense for us to have a feedback agent? Do Eclipse-based products out there already have their own feedback agent mechanisms? Would we benefit from having a standard one at Eclipse that people could build on top of?

One problem we would have with this is how to manage all the feedback. However, Mozilla must have the same problem too because they have a ton of users. Who sifts through all the feedback. Is there some processing on the backend to make sure things get sent to the right place or flagged somehow?

In the end, I think this is a good idea as it provides an communication channel between the software you write and your users. Plus, I could imagine hooking up the feedback mechanism to a Twitter feed for comedic purposes.

Thoughts?

14 Comments
  • Posted at 16:18, 2009-04-30

    Yes! I was ranting about thAT one of these days. I miss the functionality in the Eclipse SDK and other Eclipse-based products (including mine), and hate seeing multiple inconsistent ‘report an error against XYZ’ actions in the UI.

    It should support reporting a problem against a specific “product” (not sure to what Eclipse abstraction that would map to – plug-ins, features and products all have their caveats), and should not be tied to a specific bug tracking application.

    Given the need for integration with multiple bug tracking applications, maybe this should be somehow related to Mylyn?

  • Jacek
    Posted at 16:52, 2009-04-30

    there’s something along these lines in Eclipse called StatusHandling api. You can set your product to use a custom StatusHandler to do whatever it wants with an error – open Feedback window, create a bug, etc. I know at least one adopter who enhances error dialogs with additional content thanks to StatusHandling API.
    In Eclipse it’s not really used, because adopters would be unhappy to see Feedback window linking to Eclipse.org instead of their XYZ.com. Open a bug against StatusHandling (Platform/UI) to change that 🙂

  • Posted at 16:53, 2009-04-30

    The Architecture Council has been discussing this. Mylyn has the oppurtunity and feature to allow reporting a bug back to eclipse. Not sure that is exactly what you want, but how often does Eclipse crash to the point you can’t do anything? Not often in my experience.

    The Architecture Council ran out of time for the current release train to add this to every package, but I believe it is a topic for the next release train.

  • Konstantin Komissarchik
    Posted at 17:36, 2009-04-30

    I used to work on on an Eclipse-based product (BEA Workshop) that had just such a system implemented. While I still think that a good system for this style of problem reporting can be devised for Eclipse, there are quite a few obstacles and many ways to get this system tragically wrong. I will highlight some of these issues:

    1. What’s a failure? When a product like FireFox crashes, it just dies. One of the great things about Eclipse is that it rarely actually dies. Different parts and actions fail, but the product still keeps running. So how do you detect a failure event to report? What we did in Workshop is look for stack traces emitted to the log. When we detected an error log event with a stack trace, we would open a dialog informing the user of the failure and giving them a chance to send a report. The problem is that all failures are not equal in severity and some failures can come at a quite high clip with relatively little impact on what user is currently doing. In the first release of this system, we were fielding a lot of newsgroup posts from irate users who go sick of this dialog popping up in their face all the time. Some measures can be taken to mitigate this by detecting failures already reported or having rules about what can be ignored or letting users tell you to not bother them again. In the end all of these are just stop-gap measures. Eclipse environment requires a different approach. Perhaps you prompt the user to opt-in / opt-out on first failure and then proceed silently after that. You loose the ability for the user to type in comments about what they were doing, but we did not find that users were taking advantage of that feature anyway.

    2. Signal to noise ratio. A good problem report needs sufficient information so that the dev studying it can reproduce it or at least make an educated guess as to the cause of the problem. It is extremely difficult for automated systems like this to produce good problem reports. You have the stack trace, but that only takes you so far. Gathering more information doesn’t necessarily help either. We could capture plugin listings, full stack dumps, etc., but unless we have some kind of a crazy simulator, it is unlikely that the developer will be able to make sense of the problem report. Expect only a tiny fraction of reports to be actionable. We were seeing something like 5% to 10%.

    3. Separate system from bug database. You would not believe how quickly a system like this can accumulate problem reports. Even if you have a really good algorithm that can detect when two stack traces represent the same problem (not an easy thing to write without producing high rates of false positives), you should still expect to gather problems faster than your resources allow you to analyze them. There are a couple of lessons that fell out of this for us. (a) The action of turning an automated report into a bug report is something that human does during the problem triage. (b) The system needs to be able to prioritize reports somehow. A good rule is to count how many time a certain problem is reported. (c) The system needs to be able to shed old reports automatically. You really don’t care about problems reported two months ago. You have enough coming in this week to worry about.

    4. Whose problem is it? This is actually an extremely difficult question to even approximate the answer to. Is stuff closer to the top of the stack trace more at fault than the stuff lower down? Not really. Not in all cases. An approach that worked decently is to define pattern tags that would match some portion of package name to whatever level of granularity we cared to apply in a particular case. The system would tag reports automatically on arrival with however many tags applied. The individual devs would setup queries looking for certain tags. For every problem report they could unset the tag if they don’t believe the problem is in their component. If the dev looking at the problem believes there is enough to go on, we had a button that automatically opened a case in the bug database and marked the problem report as handled. An approach like this could work well for Eclipse as it scales across projects and companies. This would require a single central problem report database maintained at eclipse.org that anyone could query.

    5. Which release is it? Is it a dev build? Was there any unreleased / patched code running that may have contributed to the problem? We did not find a good answer to all aspects of this problem. It is one of the contributing factors that creates the low signal to noise ratio.

    If there is interest creating such a system at Eclipse, I would be interested in contributing to that effort, but the system needs to be built in such a way that is inherently useful to both Eclipse projects and third parties extending Eclipse.

  • Posted at 18:13, 2009-04-30

    Yes! We developed multiple RCP apps and they always had implementations of Feedback Dialogs. It’s something the IT departments in every company wants to provide for their customers.

  • Min Idzelis
    Posted at 21:05, 2009-04-30
  • Karl Matthias
    Posted at 00:15, 2009-05-01

    What? Guys, Eclispe never crashes!

  • Karsten Silz
    Posted at 02:23, 2009-05-01

    I think Eclipse should have a system similar to Netbeans where in developer builds (not end user builds, I believe) you can optionally post exception stacktraces to a bug website.

  • Bjorn Freeman-Benson
    Posted at 02:59, 2009-05-01

    Use the community: dump all the feedback into a huge open and transparent database of core dumps. Let the community filter through the dumps to find useful pieces of information. Who knows who will find the gems? It might be committers, it might be users, it might be a graduate student working on a thesis… the key is to be open and use the power of the crowd.

  • Posted at 03:38, 2009-05-01

    We have an in house RCP application, and we did some work on the client side w/ this idea. We used the StatusHandling api to detect exceptions thrown by own bundles, and opened a dialog. If the user hit ok the error is sent via http to a simple servlet and then a database. Add in a “don’t bug me again” checkbox to keep users from being annoyed. Also, we added a “grab a screenshot” option so that if the error was resulting in some weird GUI behavior the user can send that (also have a button for user submitted reports/suggestions) which can really help getting a good description.

    What I’d love to see is a) some better support inside eclipse for this – a base error reporter along w/ an extension point. Even better – would be a community (eclipse or some other place) project for a basic server-side report receiving, storage, and processing system – it should have some mechanisms for filtering out garbage, and an extension mechanism for customization (I could see it being used by the various RCP people and some eclipse downstream builds as the basis for in house servers).

  • Posted at 07:54, 2009-05-01

    Oh, what a wonderful idea!!!

    I was certainly looking forward to those days where each and every error log entry with a stack trace containing the text “cdo” somewhere automagically ends up in my bugzilla inbox!

    And the users: They’d never again have to write all these boring addon infos like “What did I do?”, “How can you reproduce?” or “I looked at the code and here is my solution!”.

    Why not become even better? What about an AutoFixMeAgent?

    😛

  • Posted at 09:11, 2009-05-04