Punctuated Equilibrium, Complexity, and Kanban

Depending on who you listen to, you might get the idea that a Kanban system might be great for simple problems, or even complicated ones, but when the going gets tough and a complex problem needs to be solved, you need something like Scrum (See Ken Schwaber’s post, and don’t skip the great comment thread…).

I don’t know if those making these statements really mean them or are using them as FUD to try to protect the Scrum brand. I am trying to figure this out for myself and thought I would share my thoughts. I will of course try to stand on the shoulders of giants in the Kanban and Complexity world like David J Anderson and Dave Snowden, and mainly summarize my understanding.

My approach here will be to lay out my understanding of the requirements from an approach that embraces Complexity, and then how I think Kanban maps to those requirements.

It seems that simple, non-prescriptive frameworks is what you need when you are working on a complex domain. These allow emergence and empirical learning. In order to learn you need feedback. In order to have effective feedback, you need to output from the system. In order to have fast learning you need fast feedback, and for fast feedback you need early and ongoing output. When I explain the Agile Manifesto to people I lay this as the main rationale for “Working Software”. You want to learn fast both about the fit of the product to the needs of the business/users, as well as the fit of the development process to the task at hand. You “Inspect and Adapt” both the Product and the Process.

This emergence seems to work best when the work is constrained in several ways. Takeuchi and Nonaka in a 1986 Harvard Business Review study entitled “The New New Product  Development Game” describe timeboxing as one constraint. When coupled with a higher purpose/energizing vision this leads to faster time to market and product innovations/creativity.

A Scrum and Complexity post at AgileEvolution.com talks about punctuated equilibrium - the equilibrium being the “Safety Zone” of working in a stable system for a while (e.g. during a Scrum Sprint when the Sprint Backlog does not shift within the sprint) punctuated by events that allow the chaos/shifting world outside to affect the system, and then return to the “Safety Zone” to have an opportunity for behaviour that fits the new reality to emerge. This has been observed in nature as well as contributing to effective evolution.

Ken Schwaber talks about the container -

“A container is a closed space where things can get done, regardless of the overall complexity of the problem. In the case of Scrum, a container is a Sprint, an iteration. We put people with all the skills needed to solve the problem in the container. We put the highest value problems to be solved into the container. Then we protect the container from any outside disturbances while the people attempt to bring the problem to a solution. We control the container by time-boxing the length of time that we allow the problem to be worked on. We let the people select problems of a size that can be brought to fruition during the time-box. At the end of the time-box, we open the container and inspect the results. We then reset the container (adaptation) for the next time-box.”

To sum up - we need Containers, Punctuated Equilibrium, and freedom for emergent behavior within these containers.

If we start from Punctuated Equilibrium - You might think Kanban doesn’t provide it since it doesn’t provide the safety zone of the sprint. But actually, if your policy is that you don’t touch what is currently in progress, you get the environment you need. There is the stability of the work in progress, and opening the hatch to the chaos on the outside whenever work is completed and we pull a new item to work on. The next items to work on can change as many times as we want. We just care about the snapshot state of the items ready to be pulled in when we open the hatch. Yes, you can say expedited items can interrupt that equilibrium. But Scrum teams deal with expedited and support items as well. Both approaches will tell you to try to shape the demand such that you reduce the amount of these interruptions, and try to create teams that can focus and achieve effective flow.

Beyond that, remember that the Kanban Method starts from your current reality and helps you see the dysfunctions, wastes and inefficiencies and provides the vision of a better reality. The specific steps that you need to take from where you are towards the effective flow vision will differ depending on your context. That is by the way part of how Kanban embraces complexity. It doesn’t prescribe a lot of practices or roles. It acknowledges your context is your context, and while there might be some good practices that might fit some situations, in complex systems you cannot even get a playbook saying this practice will get you out of this mess. You only get a catalog of practices/ideas that you might want to try out and see if they are a step in the right direction. If they are, reinforce them. If they are not, throw them away and choose something new. Evolve your system of work towards a more productive and valuable state.

If we dive into the details, one might ask what is the unit of work that is the container in Kanban. What is the equivalent of the Scrum Sprint Container? I see several options. One is to look at the level of the Business Value Increment (BVI) (or Minimally Marketable Feature, MVF, whatever you prefer to call those). When you pull in a BVI, you set a clear boundary and create a container for people to play in. They now need to deliver smaller slices of functionality until they reach a state where they can output the BVI. Within that container the functionality requirements might change adding and removing stories/tasks and the implementation will emerge.

There is nothing in Kanban that tells you what it means to be ready to start working on a BVI and what it means for a BVI to be done. You start with your current process policies, and hopefully with time you adjust your policies to get better results from the container. If you see that you waste too much time hunting for details after starting, you might try tightening up your definition of READY. If you see you are spending too much effort upstream of the work, you might want to try relaxing your definition of READY. If you get too much failure demand, try a tighter definition of DONE. You get the opportunities to affect the constraints of the container.

Another option is to look at a lower level as the container. Maybe a User Story is a better container? have a safety zone while working on the story, and look at the story boundary as the time you look outside? The BVI resonates better with me personally, but I’m not sure about it, and would love to hear what others think.

One problem with a Feature/BVI is that it’s missing the timebox. The timebox is another constraint/aspect of the container. Without it we are missing a certain pressure to be creative and innovate. On the other hand, with it we might be pressured too much and sacrifice quality or value for the sake of time. In a sterile world, the Scrum Timebox provides the pressure while allowing continued work to deliver value if the direction that emerges at the end of the timebox is seemed useful. In reality, the timebox itself sometimes provides too much pressure, leading to lower quality, unsustainable pace, or losing opportunities for value innovation.

Don Reinertsen recommends we look at Networks and Operating Systems for ideas. Modern operating systems need to deal with processes/jobs that have unpredictable duration, and still provide responsive multi-tasking. They simply cannot “trust” a process to return control to the operating system to allow another process to get some CPU time. So they use pre-emption. This means that after a time-slice called quantum, the operating system wakes up and has a chance to decide what to do. Do we keep running the current process, or is it better value for the overall system to evict it and run another process? We can use this quantum approach with Kanban as well. We can set a quantum time for each work item in progress. When that time expires we decide whether we get more value from continuing to work on it, or from finishing it up and evicting it. Applied to BVIs, it means that after a certain time, we wake up and run a semi “steering committee” for that BVI and decide whether to continue developing it, throw it away, trim the tail, or whatever else we want to do. This can add timeboxing that is BVI specific.

There is more to be said about how this BVI as the container might work, but let’s leave it to a future date. This post is running long anyhow. I also think FDD Design/Build by Feature might provide some practical examples of how this might look like.

Well, assuming you are convinced that the right Kanban system can embrace complexity, there is only one issue I’m thinking about - Not all Kanban systems will embrace complexity effectively enough. If your process has too many prescriptions and hand-offs, not enough protection of the work in process, not enough punctuation opportunities to invite chaos to pay a visit, then your Kanban system might do you a disservice.

Kanban talks about starting with what you have. Assuming what you have is not a good system for embracing complexity, what do we do to ensure Continuous Improvement / Evolution of the system is towards a direction that better embraces complexity?

Is it enough to set the compass to the “Improved Flow” North Star? Do we need to give more guidance? I’m still thinking about this, hopefully you are now too… Leave a comment and let me know what you think…

http://www.agileevolution.com/scrum-and-complexity-theory

New New Product Development Game

Kanban - When Is It Not Appropriate - David Anderson at LKBE11 (video)

http://kenschwaber.wordpress.com/2010/06/10/waterfall-leankanban-and-scrum-2/

Feature Injection - http://www.infoq.com/news/2009/05/feature-injection-comics

Dave Snowden on Cynefin

Improving flow means fewer bottlenecks, faster feedback, and less wasted capacity. Explore flow advisory or let’s talk.