Updated in April 2025: This topic was at the forefront of people’s minds in 2011. The challenges of managing the testing flow have arisen twice recently, so I took some time to refresh this post. Even in the age of Product and AI, managing the testing flow remains an important pillar. While new tools can make automated testing and deployment easier, the concept of noticing bottlenecks and working to elevate them still applies. This is how you should focus your GenAI efforts: on your bottlenecks.
Are escalating stabilization costs and unpredictable release dates giving you nightmares? You’re not alone. Many organizations, whether fully agile or taking an evolutionary approach, struggle with the testing bottleneck, especially in complex environments. Let’s dive into how flow approaches can help you manage agile testing more effectively and finally tame that testing beast!
The Challenge: Testing in Complex Environments
In today’s fast-paced world, organizations are eager to become more agile. But, as we often see, the transition isn’t always smooth. Many enterprises encounter a significant “stabilization” or “hardening” phase, which can consume around 30% of the entire release effort and schedule. This often results from high transaction costs associated with testing cycles and a desire to optimize testing efficiency by reducing cycles and pursuing scale.
The problems are clear: waste, ineffective handoffs, and rework lead to low predictability and raise serious questions about meeting release-level quality standards on time. Big-batch thinking inherent in traditional software development fuels this cycle, leading to increasingly longer stabilization phases and rushed development cycles.
Scenarios and Solutions
When it comes to agile transitions and testing, several scenarios emerge:
- Ideal Agile: Agile teams take on all stabilization work, delivering release-ready software within each increment. Massive test automation minimizes regression and stabilization efforts.
- Complex Environments: High regression effort and extensive testing coverage make it impossible for agile teams to complete all testing within a sprint. Enterprise readiness, performance, and security aspects require specialized approaches and considerable time.
- Evolutionary Agility: Organizations take a more evolutionary approach without necessarily having fully-fledged agile teams, resulting in lower “doneness” of work.
- Focus on Release Costs: Some organizations are primarily focused on reducing release costs and improving due date performance, with agility as a secondary benefit.
For complex scenarios, a flow-based recipe can significantly reduce stabilization costs and improve predictability, even without full team-level agility and engineering practices.
What’s Causing the Problem?
Late feedback on decisions is a major culprit. Late quality is always more expensive than timely quality. We often work in big batches due to perceived efficiency, but this delays feedback. Multitasking, even at the organizational level, further exacerbates the problem, as it increases the time required for features to be ready for feedback.
Ultimately, we end up “adding” quality after the fact, leading to a choice between compromising on due dates or compromising on quality. This often results in teams being driven beyond sustainable pace, ignoring bugs, and applying quick fixes, increasing unpredictability.
Working Towards a Solution
- “Done is Done”: A feature is “Done” only when it’s completely finished, passing release-grade criteria, including all test coverage and defect fixes.
- Limit Features in Progress: Allow a new feature to start only when a previous one is truly finished and release-ready with a low number of open defects.
- Optimize Features to “Done”: Developers should focus on getting features to “Done” rather than just “Coded.” This may involve investing more in quality at the coder level, such as automated unit tests, code reviews, and pair programming.
- Deal with the Testing Bottleneck: Identify work types that don’t need to go through the testing bottleneck or require less capacity from it. Choose work that is not testing-heavy or can be tested without involving the bottleneck.
- Invest to Elevate Bottlenecks: Create a backlog of engineering investments to improve testing capacity, such as test automation. Leverage the “Test Automation Pyramid” (unit testing, service/API testing, UI testing) to reduce the cost of test automation and benefit from developer involvement.
- Dynamic Stabilization Decisions: Choose when to go into stabilization based on metrics like the number of open defects and the amount of feature work in progress.
Conclusion
Escalating stabilization costs are still a common pain point. Lean/agile approaches can significantly help. By applying an end-to-end, pragmatic approach and tying all the parts together, we can drive evolutionary improvements and global optimization. This solution can be used as part of agile implementation projects, on a standalone basis, or as a precursor to adopting a more agile approach.
Ultimately, this method helps us identify bottlenecks, focus improvement efforts on these bottlenecks, improve flow, and achieve lower stabilization costs and better release flow that enables tighter feedback loops, improved time to market and convergence to valuable outcomes. It’s about working smarter, not just harder, to deliver high-quality software predictably, which is a crucial element of providing awesome products that delight your customers.
Notes
This blog post is based on a paper included in the proceedings for the Lean Systems Conference in 2011 where I presented this topic. I refreshed and reshared it on substack recently.
PS interested to see the slides from the 2011 conference talk?