QA/DEV Protocols - Opening high quality bugs

In another post in the series about QA/DEV protocols, I’ll talk about opening high quality bugs, why its important, what are the forces operating on each side of the trench here, and try to describe an approach that might improve the state of affairs a bit.

First - a definition. What is a high quality bug? To be clear, we are talking a bug report of course. The quality here refers to the accuracy of the scenario, describing exactly what is necessary to reproduce, not more, not less. It refers to providing all the auxiliary information required to analyze the bug and start working to a resolution. It also aims to report A single bug, not several issues.

It might be easier to convey the point by showcasing some examples for low quality bug reports:

• Missing logs

• Logs of different components are not time-synched, with no way to understand the time-space relationship. (This is relevant mainly for distributed systems )

• Errors happened, but are not mentioned explicitly in the bug report

• Bug report focuses on analysis, not on reporting the facts. Analysis is a bonus for QA engineers, only relevant AFTER reporting the full details.

• Much happened on the system, a couple of different scenarios, and the bug is hidden somewhere in piles of logs/information.

• Unclear bug report, leading to difficulty to prioritize and understand by business people (PM) and DEVs.

• A complex long scenario is reported while the bug is reproducable via a simple short one.

• The reported severity doesn’t match what really happened, leading to “cry wolf” or serious issues masked as trivialities.

• Multiple bugs in the same report

• Numbers - Avoid using statements like “very large” or “a lot of time”. Always include the numbers you are talking about. What seem large to you may seem small to someone else, or vice versa.

Also check out FogBugz - The Basics of Bug Tracking

Now that we have deducted what a high quality bug report is, we can try to understand the forces influencing the people opening bugs and why sometimes low quality bug reports do happen:

• When QA people find a bug, they want to report it and move on. Sometimes they feel they are metered by quantity not quality, sometimes they actually are…

• Especially for hard cases, the scenario is not that clear, and indeed there is some mix of events (including a full moon on a friday the 13th for the real nut cases) that cannot be easily reduced to a simple scenario. Trying to do this without the internal understanding of a DEV guy might take very long without being very effective.

• QA engineers are human. When the test setup/teardown is complex and requires attention to many small details (clear logs, sync time, grep for patterns in logs, etc.), things will get lost from time to time.

• In some cases, the QA group or a specific engineer is not aware of the price of low quality bug reports. (point him here…). DEV guys might not be able to put a finger on it either, or are just entrenched and prefer to point fingers and exchange emails instead of working to establish a protocol.

So what can be done?

• Discuss and educate - like I hinted, sometimes the most important step is to talk, map the expectations and root causes, and agree on a protocol, with the relevant SLAs.

• Assist QA by providing small automated snippets that can assist with test setup/teardown/analysis, guides them thru the steps to a high quality report, and really leaves them with the important step of reducing the scenario to the minimum. (btw, its possible to do the scenario reduction in automated testing harnesses as well, by retracting steps and verifying health and expected results very frequently)

• Work with very granular test cases - minimizing the scenario length. Still, combining different test cases in parallel will add complexity, but when the building blocks are small, its better than nothing.

• Issue tracking system should guide the reporter thru the important information/steps to a high quality report.

• DEVs should provide contructive feedback - when bug reports are below par quality, and when they are above. Do it privately when below quality, and publicly when above.

• Do “peer review” of bug reports when relevant - for rookie QA engineers, for difficult bugs, etc.

• In hard cases call in a DEV and get his advice on what needs to be done to make sure the report has its best chance to become a high quality one.

Any other ideas?