QA Effort Effectiveness

How do you know your QA effort is being effective ?

Based on the different stakeholders which require input from the QA a typical answer might be that Product quality is high when released to customers.

Assuming that is indeed more or less what someone expects (I’d say effective QA needs to answer to some other requirements as well) how does one go about checking whether the product quality is indeed high?

Those who reached a fairly intermediate level of QA understanding would easily point out that the percentage “QA Misses” (namely, the number of issues missed in QA and detected in the field) should be below a certain threshold. A high number here means simply that too many issues/bugs are not detected during the entire QA coverage only to be embarrassingly detected by a customer.

If one naively optimizes just for this variable, the obvious result is a prolonged QA effort, aiming to cover everything and minimize the risk. If no reasonable threshold is set, there is a danger of procrastinating and avoiding the release.
See The Mismeasure of Man of a cool article on abusing measurements in the software world…

Of course, a slightly more “advanced” optimization is to open many many bugs/issues so the miss ratio will become smaller due to the larger bugs found in QA, not due to missing less bugs. This can result in a lot of overhead for the QA/PM/DEV departments as they work on analyzing, prioritizing and processing all those bugs.
Did I forget to factor in the work to “resolve/close” those issues? NO! Several of those issues might indeed be resolved and verified/closed, but those are probably issues that were not part of the optimization but part of a good QA process (assuming your PM process manages the product contents effectively and knows how to enforce a code-freeze…).

My point is that there are a lot of issues that are simply left there to rot as open issues, as their business priority is not high enough to warrant time for fixing them or risking the implications of introducing them to the version.

A good friend has pointed this phenomena to me a couple of years ago, naming it “The Defect Junk Factory” (translated from hebrew). He meant that bugs which are not fixed for the version on which they were opened on, indicate that the QA effort was not focusing on the business priorities. The dangers of this factory is a waste of time processing them, and the direct assumption that either the QA effort took longer because it spent time on these bugs, or that it missed higher business priority bugs when focusing on these easy ones.
Kind of the argument regarding speed cameras being placed “under the streetlight” to easily catch speed offenders (with doubtful effect on overall safety), but all the while missing the really dangerous offenders.

So what can be done? my friend suggested measuring the rate of defects that are NOT fixed for that version. The higher this number, the more your QA effort is focusing on the wrong issues.
Just remember that this is a statistical measure. Examining a specific defect might show that it was a good idea for the QA to focus there, and the fix was avoided due to other reasons. But when looking across a wide sample, its unreasonable that a high number of defects are simply not relevant. If not a QA focus issue, something else is stinking, and is worth looking at in any case.

Another factor of an effective QA is fast coverage. What is fast? I don’t have a ratio of QA time related to development time. Its probably a factor of the type of changes (Infrastructure, new features, Integration work) done in the new version as each type has a different ratio of QA to DEV effort. (e.g. kernel upgrade usually requires much more QA compared to DEV effort )
Maybe one of the readers has a number he’s comfortable with - I’d love to hear.
What I do know is that version-to-version the coverage time should become shorter, and that the QA group should always aim to shorten this time further without significantly sacrificing overall quality. I expect QA groups to do risk-based coverage, automation for regression testing, and whatever measures which assist them in reducing the repeatable cost of QA coverage at the end of each version. The price/performance return on reducing the QA cycle is usually worth it to some extent.

To sum up, a good QA effort should:

• Minimize QA misses
• Minimize the defect junk factory
• Minimize QA cycle time without compromising quality

What do you think is a good QA effort? How are you measuring it?