On WontFixing Bugs

Written by Asanka Herath on  | #Engineering

This was partly inspired by a Tweet by a co-worker. The tweet is:

For projects that are under-resourced but has heavy usage it is in fact quite normal to accumulate a large number of issue reports over time. Take Chromium for example. There are (at the time of writing) ~95,000 open bugs. Many of these are task tracking bugs rather than tracking actual product bugs. But still that’s a lot of real bugs.

What’s even more interesting is the accumulation of bugs. Typically for a “successful” project the rate of new issues being reported will exceed the rate of issues being fixed. Hence the growth.

What Matters …

Issues are often annotated to indicate some measure of severity and priority. These are two different axes of measurement:

Severity refers to the degree of impact that an issue has. How many users are affected? Is the effect isolated or spread across a large user base? Severity is supposed to summarize the answer to these questions.

Priority refers to what the issue triagers deemed to be the relative importance of the issue. A bug that affects a large number of users – hence a high severity – might not be a high priority due to mitigating factors. Typically a project owner would tackle higher priority issues first. Under-resourced projects – which is the case for pretty much every successful project I’ve seen – will accumulate lower priority issues across time.

Neither severity nor priority are static either. An issue with a low severity might be elevated to a higher severity if the affected population of users is discovered to be larger than originally assumed. Or the priority may be raised if the issue is blocking the fix for another more severe issue. It’s not uncommon for some low priority issue to be suddenly elevated to critical status once someone figures out how to exploit it.

In practice perceptual priority is the only one that matters. It doesn’t matter what the issue tracker label or score is. What matters is that there’s a contributor who’s motivated to fix it. The decisions of a contributor are complex and subjective. So while the most important issues will likely get fixed, just because an issue is real or important doesn’t mean that it will be addressed.

What this means is that it is important for issue reports to stick around. At worst they will accumulate more evidence of severity or clues to what the issue is. Often the desired outcome isn’t universal either. So an issue report also functions as an opinion catchment area where interested parties can hash out their differences.

And What Doesn’t …

However, the health of the project then depends on not only handling new issues, but also in how they shepherd existing or known issues.

Much like any other todo list, the effectiveness of the issue tracker decreases when there’s an overwhelming number of items to look at. The cognitive overhead of correct prioritizing increases dramatically, as does the time it takes for triaging.

Strategies That Don’t Work …

So projects tend to use various techniques to deal with the growing pile of issues. The more overwhelming it is, the more drastic the mitigation strategy.

One such strategy in action is what we see in the Tweet above where there’s a bot that automatically closes old issues that haven’t seen any activity. At first glance this seems to make sense.

The assumption seems to be that if the issue was important enough then an endless stream of users would pipe up on the bug with “me too”s to keep the issue open. The issue tracker is now a battlefield where issue reporters have to continually work to keep the issue alive until a contributor takes pity and addresses it.

Issue reporters themselves who are often volunteers who are electing to participate in a project can also burn out. If the issues they file are automatically closed, it’s frustrating to the reporters. Chances are that with every few issues that are closed due to inactivity, the project loses a contributor.

Another strategy is to have new issues start at some sort of unconfirmed state. Once the issue has been acknowledged and accepted it can proceed to some other state which makes the issue become a candidate for contributor consideration. This one is a bit better in that it doesn’t discard confirmed issues.

However, eventually the real issues will accumulate to the point where some sort of bookkeeping is needed. Along comes WontFix and friends. This is a state where the issue is marked as “this is a real issue but we will probably never get around to fixing it.”

Reporters nevertheless find this state to be no-less frustrating than an outright closure. WontFixed bugs don’t show up by default in issue searches, so they are effectively undiscoverable to future issue reporters. All time invested in that issue is now lost.

And One That Does? …

The alternative is to just leave the issue open, but lower the priority down to something that reflects the current opinion of the project maintainers.

This strategy acknowledges that the issue is real. It serves as documentation that the product has a known defect, and in addition it can serve the purposes of accumulating evidence of severity along the way. If the issue becomes prominent enough then it can be retriaged.

That’s it. So far I haven’t seen alternatives that are much better than this. I’m generally against mass issue closing events like declaring “bug bankruptcy.” The bugs don’t care about the state of the issue report. All you’ve accomplished is to throw away the accumulated history and evidence of a defect.