Estimating bugs – why it is not worth the effort

“How to estimate bugs?” – a question many software development teams asks nowadays. Among all of the potential solutions, there are a few worth experimenting with. I would like to share with you how we deal with estimating bugs at Pragmatic Coders.

Estimating bugs

Benjamin Franklin once wrote “(…) in this IT world nothing can be said to be certain, except bugs and estimates.” erm…not quite, but if he did, that would be quite accurate. I have heard some of the development teams dropped estimating practices (salute #noestimates!) and maybe, just maybe there are products that have never had a bug. Still, bugs and estimates are what teams I worked with have had in common. And that is ok, as long as people do not mix them together. Let me tell you why estimating bugs is not worth the hassle.

Dealing with uncertainty

When estimating backlog items, teams want to gauge how much time/effort the work is worth (later used to form plans, deadlines etc.). The thing is, we cannot predict future and thus are never 100% accurate. What we can do is to improve precision by reducing uncertainty. While it is achievable with User Stories (e.g. via refinements, spikes), bugs are a different case – mainly because of higher risk (random occurrences, potentially unknown codebase, not uncovered side effects or urgency) and complexity (a lot of places to check in code, scenarios to predict). Before estimating bugs we need to do a research – sometimes there is no time for it (a critical ‘asap’ case) or we mix it up with doing an actual fix. We usually end up with no estimates, bad estimates and somewhat accurate estimates in your backlog and a lot of time consumed during a research-estimation process.

Bug is a waste

That is what I believe to be the case. Look at it this way, as development teams, we would like to ship awesome products to our customers, solve their problems, deliver value. Customers are not happy about bugs. I do not see time spent on solving bugs as a value, rather a wasted opportunity to deliver more value. What we want to do is to spend 100% of our time and efforts delivering not fixing¹. It means getting to a point where the degree of bugs in our backlog is very low. The fewer issues (waste) our software generates, the more capacity we have for adding value.

Estimate what adds value and maximize it

That brings us to the next point – we have seen that estimation is a costly and inaccurate process. Let’s use it where it can be best utilized. A simple scenario: we have a backlog filled with items bringing value (e.g. User Stories), items that do not (e.g. bugs) and estimate only the first group. We are working in fixed iterations and after a planning session end up with a set of items, some estimated some not – User Stories worth 89 Value Points and 8 bugs. We finish the iteration with 76 Value Points and 7 fixed bugs. Our velocity is 76, nothing more. You may say “Hey! But we have done a lot of bugs and spent 1/4 of iteration on them. We should count them too!”. That would be correct if our main aim was to maximize the number of bugs fixed. Our goal is to maximize the value we bring to our customers. Spending time on fixing bugs prevents us from achieving it, simple as it is.

Measure and visualize

Velocity is a useful measure in mentioned setup. It gives a good feeling how fast we deliver value (not backlog items). At Pragmatic Coders, we’ve taken this further by focusing on delivered user stories as a core productivity metric—here’s how we measure developer productivity based on value, not effort.

You can also track sum of time spent on bugs in an iteration – another practical metric. Given the example from the previous paragraph, only ~75% of the time team spent on features. We have been tracking it for a couple of iterations and see that it usually revolves around 20-30%. Now that is some useful knowledge. We are constantly spending time on fixing bugs. Maybe it is time to look at code quality or check if automated tests do their job. If you want to deal with high technical debt and there is always something more important to do – show this metric to your boss (quantify the problem) – there is a high chance fixing it may become the next priority. Measure, visualize and you will find what can be changed for the better.

¹ A great discussion on the topic. Based on it I decided to add two sentences of clarification at the end of the paragraph.