Quite a while ago at a Xen hackathon (I think the Cambridge one in 2011, has it really been 2 years?) the topic of bug tracking came up.

The developers who were present (myself included) expressed a general dislike for web based bug trackers, and in particular the existing bugzilla wasn't working for us for a variety of reasons (primarily lack of resource for care and general tending of it). They also liked the existing work flow based around mails to the Xen mailing lists, there was a feeling that this fits well with the demographics of the bugs which get reported to Xen: a sizable number of the issues are configuration errors or misunderstandings which can usually quickly be dealt with through a small email thread (followed by a request to please update the wiki). The project also gets quite a few bug reports which are initially "unactionable" due to of lack logs or a coherent description, in these cases a little bit of back and forth can quickly determine whether the reporter is going to provide the necessary additional information (this is something we are also trying to fix with clearer guidance and an ongoing effort to provide a reportbug-like tool for Xen).

However there were concerns that bigger issues which couldn't be dealt with quickly were falling through the cracks. We kicked around a few ideas for how a bug tracker that met our needs might work and that was mostly that.

Since then I've kicked the ideas around in my head a bit and searched for an existing system which would fit what was required. In the end I didn't find anything which quite fitted the bill and about six months ago I was looking for an interesting hacking project (perhaps something a bit different to my day job, hypervisor and kernel hacking) so I decided to bite the bullet, dust of my rusty Perl skills and hack something together.

Yes, that's right, I've written Yet Another Bug Tracker(tm). I confess I feel a bit dirty for doing so.

I've taken a fair bit of inspiration from debbugs (the Debian bug tracker) but I've turned the concept a little on its head. Debbugs is effectively instantiating a new mailing list for each bug report (each NNNNNN@bugs.debian.org can be thought of as a mailing list plus some additional bug specific metadata). Instead I implemented a model which takes an existing mailing list and acts as a list archive which provides a mechanism to track individual threads (or subthreads) as bugs (each bug is effectively a root Message-Id plus a little bit of metadata).

In common with debbugs bugs are created and subsequently manipulated by sending mail to a special control@ address. Commands are available to create bugs as well as to graft and prune subthreads onto existing bugs (to cope with people who break threading or offtopic subthreads etc). There is also a read-only web interface for browsing the bugs. Internally the system is subscribed to the list and records details of each message and their relationships (e.g. References headers etc) in a sqlite database. A set of CGI scripts (using the Perl CGI module) provide an interface to browse bugs.

Since the system tracks bugs based on mailing list threads I've named it Emesinae which is a "thread-legged bug" (yes, I spent longer than is strictly necessary browsing entomological websites). I've made the code available on gitorious under a GPLv2 license, see https://gitorious.org/emesinae.

I announced an alpha instance of Emesinae back in May which tracks the xen-devel mailing list. You can find the bug tracker at http://bugs.xenproject.org/xen/. Since then I've been slowly fleshing out some more useful features (such as tagging of affected branches).

It's not seen a great deal of uptake from the Xen developers yet but there are already some bugs being tracked and I'm hopeful that it will prove useful in the long run. In any case it was a fun hacking project for a few rainy evenings (and a couple more train and plane journeys).

The more I think about this, the more I like it.

I am sure you are aware, however, that many mail clients fail to do In-Reply-To/References properly. Your bug tracker would need to provide means to manipulate threads (like mutt). Ideally, there would be no other information stored about a thread than what can be obtained from the messages therein, so that when you join a message that closes an issue into an existing thread, that would yield the issue to appear closed.

Comment by madduck Fri 05 Jul 2013 18:57:28 BST

Thanks madduck.

There is a "graft" command which is intended to help address the problem of MUA's which break threading. It's not ideal (since it is manual) but I'm wary of other heuristics (e.g. subject matching) since the set of people who use MUAs which break threads overlaps a lot with the set of people who title mails "A bug" and such. There's also a "prune" command to trim subthreads which wander off-topic or to address people who hijack a thread (i.e. the people who post a new mail by hitting reply instead of new).

WRT closing, you would send a message to control@ with "close " and this message is automatically grafted onto the bug if it isn't already part of the thread, if that's what you are suggesting?

Note that you do have to cc (or better, bcc) the control address explicitly, it doesn't pick commands out of the messages received via list (which might have been what you were thinking?). I suppose it could, I've not considered it in depth, I'd be a little concerned about false positives but debbugs has similar functionality these days and appears to cope. Something to think about I reckon.

Comment by ijc Sat 06 Jul 2013 10:01:14 BST