The Kanban for Simple problems Myth
Depending on who you listen to, you might get the idea that a Kanban system might be great for simple problems, or even complicated ones, but when the going gets tough and a complex problem needs to be solved, you need something like Scrum (See Ken Schwaber’s post, and don’t skip the great comment thread…).
I don’t know if those making these statements really mean them or are using them as FUD to try to protect the Scrum brand. I am trying to figure this out for myself and thought I would share my thoughts. I will of course try to stand on the shoulders of giants in the Kanban and Complexity world like David J Anderson and Dave Snowden, and mainly summarize my understanding.
My approach here will be to lay out my understanding of the requirements from an approach that embraces Complexity, and then how I think Kanban maps to those requirements.
So what should be the behavior of an approach that embraces Complexity?
It seems that simple, non-prescriptive frameworks is what you need when you are working on a complex domain. These allow emergence and empirical learning. In order to learn you need feedback. In order to have effective feedback, you need to output from the system. In order to have fast learning you need fast feedback, and for fast feedback you need early and ongoing output. When I explain the Agile Manifesto to people I lay this as the main rationale for “Working Software”. You want to learn fast both about the fit of the product to the needs of the business/users, as well as the fit of the development process to the task at hand. You “Inspect and Adapt” both the Product and the Process.
This emergence seems to work best when the work is constrained in several ways. Takeuchi and Nonaka in a 1986 Harvard Business Review study entitled “The New New Product Development Game” describe timeboxing as one constraint. When coupled with a higher purpose/energizing vision this leads to faster time to market and product innovations/creativity.
A Scrum and Complexity post at AgileEvolution.com talks about punctuated equilibrium – the equilibrium being the “Safety Zone” of working in a stable system for a while (e.g. during a Scrum Sprint when the Sprint Backlog does not shift within the sprint) punctuated by events that allow the chaos/shifting world outside to affect the system, and then return to the “Safety Zone” to have an opportunity for behaviour that fits the new reality to emerge. This has been observed in nature as well as contributing to effective evolution.
Ken Schwaber talks about the container –
“A container is a closed space where things can get done, regardless of the overall complexity of the problem. In the case of Scrum, a container is a Sprint, an iteration. We put people with all the skills needed to solve the problem in the container. We put the highest value problems to be solved into the container. Then we protect the container from any outside disturbances while the people attempt to bring the problem to a solution. We control the container by time-boxing the length of time that we allow the problem to be worked on. We let the people select problems of a size that can be brought to fruition during the time-box. At the end of the time-box, we open the container and inspect the results. We then reset the container (adaptation) for the next time-box.”
To sum up – we need Containers, Punctuated Equilibrium, and freedom for emergent behavior within these containers.
Well, does Kanban embrace Complexity?
If we start from Punctuated Equilibrium – You might think Kanban doesn’t provide it since it doesn’t provide the safety zone of the sprint. But actually, if your policy is that you don’t touch what is currently in progress, you get the environment you need. There is the stability of the work in progress, and opening the hatch to the chaos on the outside whenever work is completed and we pull a new item to work on. The next items to work on can change as many times as we want. We just care about the snapshot state of the items ready to be pulled in when we open the hatch. Yes, you can say expedited items can interrupt that equilibrium. But Scrum teams deal with expedited and support items as well. Both approaches will tell you to try to shape the demand such that you reduce the amount of these interruptions, and try to create teams that can focus and achieve effective flow.
Emerging Process in Kanban
Beyond that, remember that the Kanban Method starts from your current reality and helps you see the dysfunctions, wastes and inefficiencies and provides the vision of a better reality. The specific steps that you need to take from where you are towards the effective flow vision will differ depending on your context. That is by the way part of how Kanban embraces complexity. It doesn’t prescribe a lot of practices or roles. It acknowledges your context is your context, and while there might be some good practices that might fit some situations, in complex systems you cannot even get a playbook saying this practice will get you out of this mess. You only get a catalog of practices/ideas that you might want to try out and see if they are a step in the right direction. If they are, reinforce them. If they are not, throw them away and choose something new. Evolve your system of work towards a more productive and valuable state.
The Kanban Container for Punctuated Equilibrium
If we dive into the details, one might ask what is the unit of work that is the container in Kanban. What is the equivalent of the Scrum Sprint Container? I see several options. One is to look at the level of the Business Value Increment (BVI) (or Minimally Marketable Feature, MVF, whatever you prefer to call those). When you pull in a BVI, you set a clear boundary and create a container for people to play in. They now need to deliver smaller slices of functionality until they reach a state where they can output the BVI. Within that container the functionality requirements might change adding and removing stories/tasks and the implementation will emerge.
There is nothing in Kanban that tells you what it means to be ready to start working on a BVI and what it means for a BVI to be done. You start with your current process policies, and hopefully with time you adjust your policies to get better results from the container. If you see that you waste too much time hunting for details after starting, you might try tightening up your definition of READY. If you see you are spending too much effort upstream of the work, you might want to try relaxing your definition of READY. If you get too much failure demand, try a tighter definition of DONE. You get the opportunities to affect the constraints of the container.
Another option is to look at a lower level as the container. Maybe a User Story is a better container? have a safety zone while working on the story, and look at the story boundary as the time you look outside? The BVI resonates better with me personally, but I’m not sure about it, and would love to hear what others think.
But aren’t we missing the Timeboxing?
One problem with a Feature/BVI is that it’s missing the timebox. The timebox is another constraint/aspect of the container. Without it we are missing a certain pressure to be creative and innovate. On the other hand, with it we might be pressured too much and sacrifice quality or value for the sake of time. In a sterile world, the Scrum Timebox provides the pressure while allowing continued work to deliver value if the direction that emerges at the end of the timebox is seemed useful. In reality, the timebox itself sometimes provides too much pressure, leading to lower quality, unsustainable pace, or losing opportunities for value innovation.
Don Reinertsen recommends we look at Networks and Operating Systems for ideas. Modern operating systems need to deal with processes/jobs that have unpredictable duration, and still provide responsive multi-tasking. They simply cannot “trust” a process to return control to the operating system to allow another process to get some CPU time. So they use pre-emption. This means that after a time-slice called quantum, the operating system wakes up and has a chance to decide what to do. Do we keep running the current process, or is it better value for the overall system to evict it and run another process? We can use this quantum approach with Kanban as well. We can set a quantum time for each work item in progress. When that time expires we decide whether we get more value from continuing to work on it, or from finishing it up and evicting it. Applied to BVIs, it means that after a certain time, we wake up and run a semi “steering committee” for that BVI and decide whether to continue developing it, throw it away, trim the tail, or whatever else we want to do. This can add timeboxing that is BVI specific.
There is more to be said about how this BVI as the container might work, but let’s leave it to a future date. This post is running long anyhow. I also think FDD Design/Build by Feature might provide some practical examples of how this might look like.
How can we improve Kanban for Complexity?
Well, assuming you are convinced that the right Kanban system can embrace complexity, there is only one issue I’m thinking about – Not all Kanban systems will embrace complexity effectively enough. If your process has too many prescriptions and hand-offs, not enough protection of the work in process, not enough punctuation opportunities to invite chaos to pay a visit, then your Kanban system might do you a disservice.
Kanban talks about starting with what you have. Assuming what you have is not a good system for embracing complexity, what do we do to ensure Continuous Improvement / Evolution of the system is towards a direction that better embraces complexity?
Is it enough to set the compass to the “Improved Flow” North Star? Do we need to give more guidance? I’m still thinking about this, hopefully you are now too… Leave a comment and let me know what you think…
Recommended Reading/References
http://www.agileevolution.com/scrum-and-complexity-theory
New New Product Development Game
Kanban - When Is It Not Appropriate - David Anderson at LKBE11 (video)
http://kenschwaber.wordpress.com/2010/06/10/waterfall-leankanban-and-scrum-2/
Feature Injection - http://www.infoq.com/news/2009/05/feature-injection-comics
Dave Snowden on Cynefin
Hi Yuval,
great post. I’d like to comment more on the latter part. What came to my mind, reading it, was that most of what you say also works on the meta level for the applied process itself. Think Toyota Kata again.
I think that without even intending so, we have found a great way to build protecting containers for processes, as well as a mechanism to gain fast feedback on process changes. Also, we have found with the Kanban (extended as it is compared to the original Toyota Kanban) a great way to describe ‘target states’ states to strive for.
That, I think, is why actually Kanban works as a change agent. This is precisely what is not given in Scrum, even more so – by injecting fear of Scrum-But rather than promoting change in teams, at a certain level change might mean not doing the right thing anymore. This to me seems like a fundamental difference, which is not that relevant on the level of ‘working on items’ but on emergent change.
This, I think, is the most relevant aspect of constraining and protecting the process through Kanban and letting it change in a seemingly unsteered (the team does not have the solution for the target condition yet) but safe way.
Thanks for the post, all the best
Markus
You raise many good questions and concerns regarding software development. I know that many ‘traditional’ managers are most concerned about getting work done on time. When they think about using an agile development approach, they want to know how time is managed.
Scrum provides a safety net in the form of Sprint time-boxes, as you point out. Sprints aren’t perfect and many problems can arise but at least time is being managed directly.
Kanban presents a more complex solution. There are no time-boxes as such. There are tasks, queues and workflows. We know that we can use task time estimates and queues to help us manage time. Unfortunately, this is more complex than merely setting an end date for a Sprint and seeing what the status on the project is on that date.
But, end dates also work in Kanban. The team can set a goal to deliver a build or incremental release every 2 weeks, for example. Whatever is done at that time is delivered. Anything not done is queued up for the next delivery.
Kanban is more complex than Scrum but it is still simpler than any waterfall approach I’ve ever used. We need to do a better job of educating managers on how to use Kanban effectively.
Yuval, thank you for your excellent post! I especially liked the “quantum time” analogy that you quoted Don Reinersten as using in comparing CPU time slicing to the time slicing on an Agile project.
Whether it’s XP, Scrum, Kanban, or some other Agile flavor, the one common time slice that all Agile methodologies, frameworks, and methods seem to have in common is the “work day”. This time slice is demarcated by the “daily standup meeting”. I have yet to hear of any Agile teams that disagree with the utility of the daily standup, so I believe this might be the atomic unit of measure that you may be looking for in your search for a “safety container” in Kanban.
I view the s/w dev work in a 3-level hierarchy: MVF, Story, and Task. I look at stories and MVFs simply as rollups of tasks. My assumption is that if there’s steady daily progress on the tasks of a story (which also implies steady progress of MVFs), then that story is “under control” and need not be discussed much (or at all) at the standup.
However, if there’s no “significant progress” reported about a task that’s in-progress on any given workday, then I think that’s like when Reinersten’s quantum time expires – that’s where the CPU’s interupt algorithm kicks in and looks at any race conditions or deadlocks. i.e. That’s the opportunity for the entire team to pause for a moment; visualize the work-in-progress on the story/task; ask what’s going on; see if there’s any way to expedite the work to reduce cycle time (or break it down further to improve the visibility of the work to the team); and if there’s not, then let it continue for just one more day.
The team policy is that tasks should be broken down into work that can be completed in a day or less, so if there’s a full day gap between when a task is started and when it’s done, then that’s an opportunity for the team to identify the current state of the story/task as a bottleneck and provide some quick advice to whomever is working on it. In other words, the TEAM takes the responsibility for improving the flow of the work, rather than letting any individual (or pair of team members) proceed/flounder for more than a day without any assistance or accountability. “Yesterday’s work” is the basic atomic unit of measure (the container within which no external interupts should occur) for the individual(s) working on the task.
The standup meeting is where the rubber meets the road, and other weekly, bi-weekly, or monthly Kanban meetings are mere rollups of the daily cycle. In Scrum, the same could be said of Sprints.
Also, I encourage JIT planning of tasks (it’s common to not know the solution to a technical problem until you’ve dove into it), so even if the individual(s) were unable to break down their work into small enough chunks to fit in a day or less when they started the story, they can and should adjust their plan by adding new tasks as they arise. These new tasks reflect the new reality of the story, so that the team can visualize their work more readily; give them their one-day safety net (container); but enforce the policy of reporting progress on the story each and every workday.
Kirk Bryde, CSM, CSP, PMP
Thank you for your comment Kirk!
I agree with most of your points. I especially like JIT planning of tasks. My problem with the bottoms-up rollup you describe is that there IS a chance that although there looks to be busy-bees making progress, if there aren’t real functional stories being delivered, I will be worried. And also if there are stories delivered but the MVF is not anywhere in sight, again a reason to be worried. So while flow at the bottom levels is a necessary start, it is not enough.