Tag Archives: RND

Edible versions – Tips for implementation

Estimated Reading Time: 1 minute

So you read my Edible versions post and want to get the good stuff on how to make it happen in your organization. Well, to be honest, its not that difficult once all the parties sit together, talk about their expectations and design the protocols between the groups. See my earlier post for some general pointers.

Having said that – Maybe I CAN provide some tips that I saw working in the past:

  • Ensure all content in a delivery is tracked as a change request (bug/feature/other) in an issue tracker.
  • Provide an “Impact Level” for each change, so QA can easily focus on the high impact changes first.
  • For complex changes or large builds get used to hold delivery meetings where the DEV and QA discuss the changes and exchange ideas on how to proceed with covering this build. Be effective – know when the changes are small and the process can be lighter.
  • Try to establish an environment which automatically generates release notes for a version. At a minimum, as a report on whatever the issue tracker says. If possible, it should be based on actual deliveries to the SCM system. Use something like the integration between JIRA and Quickbuild/LuntBuild

Edible versions

Estimated Reading Time: 1 minute

All you QA people out there –

  • How often does your QA group “choke” on versions delivered by the development group?
  • Are you used to “unedible” versions which just don’t taste right?
  • How about versions which simply come as a blackbox where you have no idea what changed, therefore no idea what to do with the version, what to expect of it?

Now all of you DEV people – think about the times where you installed 3rd party products/updates which caused you the same digestion problems…

Those unedible deliveries cause a variety of problems. Lets start with the fact that whoever gets the delivery wastes a lost of time chewing it up, in the meantime not only delaying coverage of the new delivery but also NOT making progress on previous deliveries (the classic question of when to commit your QA organization to a new build delivered by R&D; and risk coverage progress for the earlier but known build. Especially tasteful when delivered to your plate a few hours before the weekend)

When the contents are unclear, QA people can only do general coverage, and the time it takes to verify regression concerns and make sure whatever we intended to fix was indeed fixed grows longer.

What is the point here? same as a sane person would refuse to swallow unmarked pills coming from unmarked bottles, refuse to take a version/build/delivery that is not documented sufficiently. I’m not aware of many good reasons not to mandate Internal release notes.

And DEV guys – consider some dogfooding on each delivery, even if its “just” to the friendly QA people next door. Lots of work you say? Well then, time to introduce Continuous Integration and Smoke Testing…

Severity and Priority – The Debate

Estimated Reading Time: 6 minutes

There are a couple of alternatives for managing severity and priority in the Issue Tracker.
Although there are many resources out there on this subject (see http://del.icio.us/yyeret/priority_severity) I’ll try to consolidate them and provide my 2c on the matter, as I think its an important subject.

Single-field Priority
First, seemingly simpler alternative, is Single-field priority – representing Customer Impact
The idea here is to only have a single priority/severity field. The reporter assigns it according to his understanding of the customer impact (severity, likelihood, scenario relevance, etc.). Product Management or any other business stakeholder can shift priority according to current release state, his understanding of the customer impact according to the described scenario. The developers prioritize work accordingly.
The strength of this approach is in its simplicity, and the fact that several issue trackers adopt this methodology and therefore support it better “out of the box”.
The weakness is that once in the workflow the original reasoning for the priority can get lost, and there is no discerning between the customer impact and other considerations such as version stability, R&D; preferences, etc.

An example why this is bad? Lets say Keith opened bug #1031 with a Major priority. Julie the PM later decided that since there is some workaround and we are talking just about specific uses of a feature rarely used, the business priority is only Normal or Minor. Version1 is released with this bug unresolved. When doing planning for Version2 Julie missed this bug since its priority is lower.
Even if the feature its related to is now the main focus of the version. Even if not missed, looking for this bug and understanding the roots and history is very hard, especially consider the database structure of issue trackers. History is available, but its not as easy as fields on the main table of issues…

Brian Beaver provides a clear description of this approach at Categorizing Defects by Eliminating “Severity” and “Priority”:
I recommend eliminating the Severity and Priority fields and replacing them with a single field that can encapsulate both types of information: call it the Customer Impact field. Every piece of software developed for sale by any company will have some sort of customer. Issues found when testing the software should be categorized based on the impact to the customer or the customer’s view of the producer of the software. In fact, the testing team is a customer of the software as well. Having a Customer Impact field allows the testing team to combine documentation of outside-customer impact and testing-team impact. There would no longer be the need for Severity and Priority fields at all. The perceived impact and urgency given by both of those fields would be encapsulated in the Customer Impact field.

Johanna Rothman in Clarify Your Ranking for System Problem Reports talks about single-field risk/priority:

Instead of priority and severity, I use risk as a way to deal with problem reports, and how to know how to fix them. Here are the levels I choose:
o Critical: We have to fix this before we release. We will lose substantial customers or money if we don’t.
o Important: We’d like to fix this before we release. It might perturb some customers, but we don’t think they’ll throw out the product or move to our competitors. If we don’t fix it before we release, we either have to do something to that module or fix it in the next release.
o Minor: We’d like to fix this before we throw out this product.

bugzilla-style Severity+Priority

Here, the idea is to use a severity field for technical risk of the issue, and a priority field for the business impact. The Reporter assigns severity according to the technical description of the issue, and also provides all other relevant information – frequency, reproducability, likelihood, and whether its an important use-case/test-case or not. Optionally, the Reporter can assign priority based on the business impact of the issue to the testing progress. e.g. if its a blocker to significant coverage, suggest a high priority. If he thinks this is an isolated use case, suggest a lower priority. A business stakeholder, be it PM, R&D; Management, etc. assigns priority based on all technical and business factors, including the version/release plan. Developers work by Priority. Severity can be used as a secondary index/sort only.
Developers/Testers/Everyone working on issues should avoid working on high-severity issues with unset or low priority. This is core to the effectiveness of the Triage mechanism and the Issue LifeCycle Process
Customers see descriptions in release notes, without priority or severity. Roadmap communicated to customers reflects the priority, but not in so many terms.

Strengths of this approach are:
* Clear documentation of the business and technical risks, especially in face of changing priorities.
* Better reporting on product health when technical risk is available and not hidden by business impact glasses.
* Less drive for reporters to push for high priority to signify they found a critical issue. It’s legitimate to find a critical issue and still understand that due to business reasons it won’t be high priority.
* Better accommodation of issues that transcend releases – where the priority might change significantly once in a new release.

The weaknesses are that it’s a bit more complex, especially for newbies, and might require some customization of your issue tracker, although if your tool cannot do this quite easily, maybe you have the wrong tool…
In addition, customers have trouble understanding the difference between their priority for the issue and the priority assigned within the product organization. The root cause here is probably the lack of transparency regarding the reasoning behind the business priority. I’d guess that if a significant part of the picture is shared, most customers would probably understand (if not agree) with the priorities assigned to their issues. Its up to each organization to decide where it stands on the transparency issue. (see Tenets of Transparency for a very interesting discussion on the matter in the wonderful weblog of Eric Sink)

To see how our example works here – Keith will open the bug, assign a major severity, and a low priority since the bug blocks just one low-priority test case. Julie the PM sees the bug, and decides to assign a low priority value, so the bug is left for future versions for all practical matters. When planning V2 Julie goes over high-severity issues related to the features under focus for the version, and of course finds this issue as it’s a Major severity.

See the following resources for this approach:
* http://c2.com/cgi/wiki?DifferentiatePriorityAndSeverity
* Priority is Business; Severity is Technical:

business priority: “How important is it to the business that we fix the bug?” techn
ical severity: “How nasty is the bug from a technical perspective?” These two questions sometimes arrive at the same answer: a high severity bug is often also high priority, but not always. Allow me to suggest some definitions.

Severity is levels:
o Critical: the software will not run
o High: unexpected fatal errors (includes crashes and data corruption)
o Medium: a feature is malfunctioning
o Low: a cosmetic issue

Priority levels:
o Now: drop everything and take care of it as soon as you see this (usually for blocking bugs)
o P1: fix before next build to test
o P2: fix before final release
o P3: we probably won’t get to these, but we want to track them anyway

* Corey Snow commented on Clarify Your Ranking for System Problem Reports:
Comment: Great subject. This is a perennial topic of debate in the profession. The question at hand is: Can a defect attribute that is ultimately irrelevant still serve an important function? Having implemented and/or managed perhaps a dozen different defect tracking systems over the years, I actually prefer having both Priority and Severity fields available for some (perhaps) unexpected reasons. Priority should be used as the ‘risk scale’ that the author describes. 3 levels, 5 levels, or whatever. Priority is used as a measure of risk. How important is it to fix this problem? Label the field ‘Risk’ if that makes it more clear. Not so complicated, right? So what good is Severity? Psychology! Its very existence makes the submitter pause to consider and differentiate between the Priority and Severity of the defect. In other words, without Severity, the submitter might be inclined to allow Severity attributes to influence the relative Priority value. Example 1: Defect causes total system meltdown. Only users in Time Zone GMT +5.45 (Kathmandu) are affected on leap years. There is one user in that time zone, but there is a manual workaround, and a year to fix it besides. Priority=Super Low, Severity=Ultra High Severity gives a place for the tester to ‘vent’ about their spectacular meltdown, without influencing the relative Priority rating. Example 2: Defect is a minor typo. Typo in on the ‘Welcome to Our Product’ screen, which is the first thing every user will see. Priority=Ultra High, Severity=Super Low Again, Severity gives a place for the tester to express how unimportant the defect is from a functional perspective, without clouding their Priority assessment. I once managed a defect tracking system with only a Priority field. This frequently led to a great deal of wasted time in defect discussion meetings as one side would argue about Severity attributes while another would argue about Priority attributes, but the parties were not even aware of the distinction that was actually dividing them. Having both fields serves to head off this communication problem, even if Severity is completely irrelevant when fix/no fix decisions are actually made. ~ Corey Snow (03/11/03)

Author’s Response: Corey, Great counterpoint to my argument. ~ Johanna Rothman (03/12/03)

Personal Favorite
As can probably be understood by now, My personal favorite is the Severity+Priority approach. I confess I don’t have much experience with the single priority approach, but I really feel the Severity+Priority way is very effective, without significant costs once every stakeholder understands it.

What is your favorite here?

QA/DEV Protocols – Opening high quality bugs

Estimated Reading Time: 3 minutes

In another post in the series about QA/DEV protocols, I’ll talk about opening high quality bugs, why its important, what are the forces operating on each side of the trench here, and try to describe an approach that might improve the state of affairs a bit.

First – a definition. What is a high quality bug? To be clear, we are talking a bug report of course. The quality here refers to the accuracy of the scenario, describing exactly what is necessary to reproduce, not more, not less. It refers to providing all the auxiliary information required to analyze the bug and start working to a resolution. It also aims to report A single bug, not several issues.

It might be easier to convey the point by showcasing some examples for low quality bug reports:

  • Missing logs
  • Logs of different components are not time-synched, with no way to understand the time-space relationship. (This is relevant mainly for distributed systems )
  • Errors happened, but are not mentioned explicitly in the bug report
  • Bug report focuses on analysis, not on reporting the facts. Analysis is a bonus for QA engineers, only relevant AFTER reporting the full details.
  • Much happened on the system, a couple of different scenarios, and the bug is hidden somewhere in piles of logs/information.
  • Unclear bug report, leading to difficulty to prioritize and understand by business people (PM) and DEVs.
  • A complex long scenario is reported while the bug is reproducable via a simple short one.
  • The reported severity doesn’t match what really happened, leading to “cry wolf” or serious issues masked as trivialities.
  • Multiple bugs in the same report
  • Numbers – Avoid using statements like “very large” or “a lot of time”. Always include the numbers you are talking about. What seem large to you may seem small to someone else, or vice versa.

Also check out FogBugz – The Basics of Bug Tracking

Now that we have deducted what a high quality bug report is, we can try to understand the forces influencing the people opening bugs and why sometimes low quality bug reports do happen:

  • When QA people find a bug, they want to report it and move on. Sometimes they feel they are metered by quantity not quality, sometimes they actually are…
  • Especially for hard cases, the scenario is not that clear, and indeed there is some mix of events (including a full moon on a friday the 13th for the real nut cases) that cannot be easily reduced to a simple scenario. Trying to do this without the internal understanding of a DEV guy might take very long without being very effective.
  • QA engineers are human. When the test setup/teardown is complex and requires attention to many small details (clear logs, sync time, grep for patterns in logs, etc.), things will get lost from time to time.
  • In some cases, the QA group or a specific engineer is not aware of the price of low quality bug reports. (point him here…). DEV guys might not be able to put a finger on it either, or are just entrenched and prefer to point fingers and exchange emails instead of working to establish a protocol.

So what can be done?

  • Discuss and educate – like I hinted, sometimes the most important step is to talk, map the expectations and root causes, and agree on a protocol, with the relevant SLAs.
  • Assist QA by providing small automated snippets that can assist with test setup/teardown/analysis, guides them thru the steps to a high quality report, and really leaves them with the important step of reducing the scenario to the minimum. (btw, its possible to do the scenario reduction in automated testing harnesses as well, by retracting steps and verifying health and expected results very frequently)
  • Work with very granular test cases – minimizing the scenario length. Still, combining different test cases in parallel will add complexity, but when the building blocks are small, its better than nothing.
  • Issue tracking system should guide the reporter thru the important information/steps to a high quality report.
  • DEVs should provide contructive feedback – when bug reports are below par quality, and when they are above. Do it privately when below quality, and publicly when above.
  • Do “peer review” of bug reports when relevant – for rookie QA engineers, for difficult bugs, etc.
  • In hard cases call in a DEV and get his advice on what needs to be done to make sure the report has its best chance to become a high quality one.

Any other ideas?

QA/DEV Protocols – Calling developers to the lab

Estimated Reading Time: 3 minutes

I’m going to dedicate a couple of posts to the relationships between QA and Development (DEV) organizations.

Anyone who’s ever been in either of those organizations knows that sometimes there seems to be a conflict of interest between QA and DEV, which can lead to friction between the groups and the people. Obviously when both organizations are running under the same roof, there must be some joint interest/goal, but the challenge is to identify the expectations of each group in order to work toward their goal and accomplish their mission effectively.

The difficult cases are those that put more strain on one party, in order to optimize the effectiveness of another. Example – developers are asked to unit/integrate/system test their software before handing over a build to QA. Some developers might say that this is work that can be done by QA, and their time is better spent developing software. The QA engineers will say that they need to recieve stable input from the DEVs in order to streamline the coverage progression, and that the sooner issues are found, the lower the cost to fix them.

One way to look at these “protocols” between the groups is via the glasses of TOC (Theory of Constraints), identify the bottlenecks of the overall system/process, and fine-tune the protocol to relieve the bottleneck. People in those groups, and especially the leaders, should be mature enough to know that sometimes doing the “right thing” might be to take on more work, sometimes even not native work for their group.

One example is the issue of when to ask DEV guys to see problems the QA engineers have discovered.
Reasons for calling DEV might be:

  • Wish to reopen a bug
  • Bug was reproduced and a developer was interested to see the reproduction.
  • New severe bug

There are a couple of forces affecting this issue:

  • QA wishes to finish the context of the specific problem/defect, open the bug, and get on with their work.
  • DEV wishes to finish the context of their specific task, and wish to avoid the “context switch” of looking at the QA issue.
  • In general, both QA and DEV have learned to wave the “Context Switching Overhead” flag quite effectively. (A more pragmatic conclusion is that some context switching overhead is unavoidable, and sometimes the alternative is more expensive…)
  • In some cases, “saving” the state of the problem for asynchronous later processing by DEV is difficult or takes too many resources to be a practical alternative.

A possible compromise between all those forces is to define some sort of SLA between the groups, stating the expected service provided by DEV to QA according to the specific situation (Reopen, Reproduced, New Severe, etc.). This SLA can provide QA a scope of time they can expect answers in, without feeling they are asking for personal favours or “bothering” the DEVs. The developers get some reasonable time to finish up the context they were in without feeling they are “avoiding” QA. The SLA can also cover the expected actions to be taken by QA before calling in the DEV, or in parallel to waiting for them. This maximizes the effectiveness of the DEV person when he does free up for looking at the issue, while better utilizing the time of the QA while waiting. (for example – fill the bug description, look for existing similar bugs, provide the connectivity information into the test environment, log excerpts/screenshots, etc.)

Another question is who to call on when QA needs help. The options here depend on the way the DEV group/teams share responsibility on the different modules of the system.

  • In case there are strict “owners” for each module, and they are the only ones capable of effectively assisting QA, the only reasonable choice is to call on them… this requires everyone to always be available at some level.
    I have to say though that I strongly advise against such an ownership mode. Look at code stewardship for a better alternative (in my oppinion) and see below how it looks better for this use case and in general…
  • In case there is a group of people that can look at each issue, one alternative is to have an “on call” cycle where people know they have QA duty for a day/week. In this case there will be issues which will require some learning on their part, and perhaps assistance from the expert on a specific area. This incurs overhead, but is worth its price in gold, when the time comes and you need to support that area in real time, need to send the DEVs to the field to survive on their own, or when the owner/expert moves on…

To sum up, like in many h2h (human-to-human) protocols, understanding the forces affecting both sides of the transaction is key to create a win-win solution. A pragmatic view trying to minimize the prices paid and showing the advantages of the solution to both sides and to the overall organization can solve some hard problems, as long as people are willing to openly discuss their issues and differences. I’ve seen this work in my organization, hopefully it helps others as well.