Guest Post – 5 Guiding Principles To Using Agile In Research-Intensive Software Environments

Estimated Reading Time: 5 minutes

Agile Meets Research / Data Scientists

As agile spreads wider and wider I often get to work with researchers (a.k.a Data Scientists) working closely with product development. When going agile these people struggle to figure out how it fits their unique style of work. One of those researchers I encountered on an Advanced Analytics group in Intel was Shahar. We had a chat recently and I asked him if he would be so kind to write a guest post describing his perspective. And he delivered! If you’re a researcher trying to make agile work or you’re implementing agile and you’re trying to help your researchers figure it out, this should be interesting!

Continue reading “Guest Post – 5 Guiding Principles To Using Agile In Research-Intensive Software Environments”

Waste Deep in Being Done – Or…Why it’s Shorter to Take Longer – Guest Post by Benjamin Spector

Estimated Reading Time: 5 minutes

Introduction

I met Benjamin Spector in one of the recent Agile Boston meetings. He told me a story that I liked a lot since it brought to life one of the key concepts I presented – The Transaction Cost vs Cost Of Delay curve (from Principles of Product Development Flow by Reinertsen). I was able to persuade Benjamin to write up this story…

Waste Deep in Being Done – Or…Why it’s Shorter to Take Longer 

“We should have finished a month ago.”  That was the item with the most votes during the team’s last sprint retrospective.  This last sprint completed the final production-ready feature the team had been working on.  It was delivered just in time for the scheduled annual major product release.  Everyone was decompressing from having faced the possibility of delivery failure.  But even as we celebrated our success, there was a sense of disappointment that we had taken as long as we did.

I had been the team’s scrum master for about 6 months, starting up with them as this latest project began.  The team was small with 4 developers (3 full-time software engineers and 1 QA specialist) plus a product owner and a scrum master…me.  When I started working with them, the team was practicing scrum at a beginner-level.  My initial focus was mainly on getting the team to perform basic agile practices more effectively for estimating and planning, as well as for daily standups, sprint reviews and retrospectives.

Over the course of the project the team was dealing with regression failures that came back to them 1-2 weeks, or longer after their original code submissions.  The problem was not terribly taxing on the team in the early stage of the project.  We’d get 1 or 2 regression failure bug tickets and just included them in the next sprint backlog to fix them.  Sometimes we didn’t get to fixing a regression failure until 2 sprints down the road.  It didn’t seem like any harm to kick it to another future sprint.  It was tolerable…or so it seemed.

The team’s practice was to submit production-ready code after running a suite of “smoke test” regression tests.  The product itself was a complex CAD tool with over a million lines of code and up to 25,000 separate automated regression tests.  Running the full suite of tests was an overnight process.  Whereas, running a smaller subset of selected regression tests that focused mainly on the part of the code base the team worked in was a common practice among all our scrum teams.  It allowed for quicker turnaround. In general, it was felt that running the regression “smoke test” suite enabled everyone to deliver quicker at a relatively low risk to the product quality.  If a couple of regression failures slipped through the net, no one thought it was a big deal.  In fact, this practice was explicitly called out as part of my team’s definition of done for user stories.

But, the frequency of regression failures began to increase.  As we got closer to the release deadline, there were more regression bugs to fix, and the time spent fixing them consumed a greater portion of the team’s capacity.   As the scrum master, this issue did not go unnoticed by me.  I wrestled with the question of when it would be the best time to raise it to the team.  We were within striking distance of our goal and the team was focused on finishing the project and complying with all the acceptance criteria for the project.  Significantly, one of those criteria was delivery with zero regression failures.

About a week before we finished the project, I began reading Jeff Sutherland’s latest book “Scrum: The Art of Doing Twice the Work in Half the Time.”  I came to the chapter called Waste is a Crime, and a section called Do It Right the First Time, and the words leapt out the pages.  Sutherland gives a powerful example with Palm Inc.’s research on the amount of time taken to fix bugs not found and fixed right away (page 99).  The research showed that it took 24 times longer to fix a bug a week or more after code submission, than if it was found and fixed at the time the developer was actively working on the code, regardless of the size or complexity of the defect.  24 times!

So there we were at the end of the project, with everyone experiencing the elation and relief of a successful delivery mixed with a sense of disappointment that we did not finish as quickly or as cleanly as we had expected.  “We should have finished a month ago.” Why didn’t we?

It was at this moment that I jumped up and said, “hang on just a second.  I’ve got to get something at my desk that will be really interesting for everyone to hear.”  I bolted out of the conference room, ran to my desk grabbing the Sutherland book, and returned slightly breathless with the page opened at the passage describing the Palm Inc. research about bug fixing taking 24 times longer.  The gist of the Palm Inc. story was about one and half pages long, so I asked the team’s permission to read it aloud before we continued with our retrospective discussion.  Everyone agreed with some amusement and curiosity about what I was up to.  When I finished reading the passage, I could see the impact in the eyes of every team member.  Each member of the team began looking at each other recognizing this shared insight.  That’s the moment when I knew I had their attention.

I put the question to the team, “How many regression bugs have we fixed since we started this project?”  The answer was 35.  I had already sized the problem when I began monitoring it closely over the last 3 sprints.  I quickly showed them my query in the Jira database, displaying the search criteria and the tally of regression bugs on the conference room overhead projector.  Everyone agreed that it was a valid.

Then I asked, “On average, how long does it take us to fix a regression bug?”  We started opening up individual records so we could see our tasked-out estimates for each one.  Examples ranged from 8 hours to 16 hours typically including tasks for analysis, coding, code review, adjustment of some of the tests themselves to accommodate the new functionality, submission for regression testing and final validation.  Some took a little more time.  Some took a little less.  After a few minutes of review, the team settled on the figure of 12 hours or work per regression bug.  So, I did the simple arithmetic on the white board: 35 x 12 = 420 hours.  Then I applied the “24 times” analysis: 420 / 24 = 17.5.  I said, “If the rule holds true, then if we had fixed the regression bugs at the time they were created, in theory it would only have taken only 17.5 hours to fix them, not 420 hours.”  Then I doubled the number just to make everyone a little less skeptical.  35 hours seemed more reasonable to everyone.  Nevertheless, it was still a jaw-dropping figure when compared with 420 hours.  While I stood pen in hand at the white board, everyone on the team sat in stunned silence.  While they were absorbing the impact of this new insight, I took to the whiteboard again and wrote down 420 – 35 = 385 hours.  Then I reminded them of our sprint planning assumptions. “Based on our team’s capacity planning assumptions, we plan for 5 hours per day per person for work time dedicated to our project.  For the 4 of you that equals 100 hours per week of work capacity.”  I completed the simple arithmetic on the white board showing 385 / 100 = 3.85 weeks, underlining and circling the 3.85 weeks.  Then I pointed back to the retrospective item with the most votes, I said, “There’s your lost month.”

When our retrospective ended we left the meeting with a significant adjustment to our team’s definition of done.  We replaced the “smoke test” regression testing requirement with the practice of always running the full regression test suite on the code submitted for a story and resolving all regression failures before considering the story done.  This change was made with the enthusiastic and universal agreement of every team member.  Everyone recognized that it would take longer to finish each story up front.  But, they were happy to accommodate the extra time because now we all knew, without even the slightest doubt, that even though it would take longer to finish the story the first time, it was always going to take a lot less time than having to go back and really finish it later.

About Benjamin

Benjamin Spector
Benjamin Spector

Benjamin Spector has worked as a software product development professional and project manager for over 20 years.  For 4 of his last 9 years at Autodesk, Inc., he has worked as a scrum master for several teams and as a full-time agile coach introducing and supporting agile practices throughout the organization. Reach out to him on Linkedin

 

Sensing and increasing manager engagement during an agile change initiative – guest post by Yaki Koren

Estimated Reading Time: 4 minutes

Earlier this week I published a guest post about how managers need to change if they want agile to succeed by Yaki Koren. Some blog/twitter followers asked for elaboration and Yaki was gracious enough to comply. I suspect the fact that this is a really hot topic for him this week helped. Without further a due, here is Yaki with some explanations of what the coaching team in his organization does to sense and increase manager engagement in order to improve agile change depth and stickiness… (Note Yaki doesn’t emphasize it but we are talking about a Kanban Method style of change initiative here which affects the kind of activities going on).
I was asked after the earlier post how do we engage managers and making sure they captain the ship. Here is a try to explain what we do (or at least what think we should be doing and when we do it, it works well).

As we are internal consultants it is quite easy to engage us – there are almost no bureaucratic barriers. We have developed a few techniques to protect managers from harming themselves by going down a dangerous path (for them).

As mentioned above we realized that a key success factor for agile implementation is a manager’s willingness to captain the ship. So the first thing we make sure is that the manager contacts us and not the other way around. We may do marketing, we may do an elevator pitch but the manager should call us, the manager should set the meeting. It’s a “call us, we don’t call you” thing. At the meeting we let the manager run it, state their need etc.

We keep doing this all through our engagement with the manager. We are making sure someone is pulling us and no pushing is done. When they stop pulling we, again, may do marketing and may encourage them but they need to take the action. If it’s not important enough for them they won’t do the action.

So, many times there are initial calls that die off and there may be a session or two and then it dies. It better die early when not too much damage was done.

Our main problem are cases where for some reason the process did start to actually run and the group moved to Kanban and then they stop pulling. These are sad cases with bad public relations, though I must say that even then it usually works better than the alternative.

Another thing we do when starting is asking the manager to budget us. We didn’t do this initially (we got the budget from top management) and when you don’t pay you sometimes don’t care. So now we’re asking for some budget, usually higher than what we actually need – and this proves to be a good test for the manager, another sanity check for their willingness to invest in the change.

When they lead it and agree to budget us we make sure they understand what are the big challenges they are going to face:

· Progress monitoring is quite different than what they’re used to – we tell them they will feel a loss of control at the beginning

· They need to invest more in day to day management – less status reviews and more execution (some managers don’t understand it: for them the change is easy)

· The process of work will keep changing. There is no winning formula we can give them – They will start with something and need to constantly adapt. It’s not a one-time bang thing. I should write a post on this aspect.

Since our organization is big sometimes we are not contacted by the person we believe is the one who should actually lead the change. Sometimes it may be a subordinate, their manager or even a peer. So they may set the meeting, they may even budget us but still we need the person who should lead this to actually show they’re serious.

This is more tricky.

You may find yourself in a room with a person whose manager asked him to do the change and the manager is not there at all. If they are not interested you are trapped. Suddenly it is you that wants this and you may find yourself pushing instead of being pulled.

What we do (or more accurately “try to do, really want to do, plan to do and sometimes it actually works”) is again to make sure this person wants it. We sit there, quietly. Many times they will expect us to lead this – however this is not ours to lead, right? So we may ask politely how can we help. You will get a deep frown doing this. And, by the way, it is fine if they tell you they’re there because their boss told them to. Then you need to understand together what is it they’re boss wants them to do. If they don’t understand why they’re there they need to get back to whoever arranged this to understand the reason.

Many times we have sessions with the project team. I like being in the center of things, giving a great performance, but it seems to be much more effective if the manager does this. I need to fight my ego and then do a good preparation with the manager (after explaining why they should do it) for the session. The more the managers lead the more effective it will be.

When we start the process we sometimes find we are becoming part of the operation. To avoid this we make sure we don’t attend all project meetings on purpose, we make sure nothing is dependent on us. This is the team’s thing, not ours. We are there to help, not do some of the job. Again, it is very tempting to do it – create the excel, lead the session, build the board – but every time we do this we skip a learning opportunity for the project team and increase their dependency on us. It is a matter of empowerment.

I believe there is more stuff but these are our main techniques for making sure only managers who are apt for the move to agile actually do it and when they do it to make it their thing, not ours.

The downsides of agile – guest post by Yaki Koren

Estimated Reading Time: 2 minutes

This is a guest post by Yaki Koren who is the lead coach in one of AgileSparks’s clients – Amdocs Delivery who are going through a very interesting transformation leveraging the Kanban Method, crossing the chasm techniques as well as several key agile practices. It is also where I spend a considerable slice of my time the last year trying to help make this happen. And now – here is Yaki…

 

Last May I gave a talk at Agile Israel about the implementation we did in my company. At the end of the talk a guy from the audience asked what is the down side to agile. I must admit I never thought of it as it looked quite perfect to me. Eventually I answered that managers who like to micro manage will find the move to agile difficult.

 

Now after more than a year of implementation I think I have a better answer to this question.

 

We are doing the transformation to agile in many projects. Some succeed more and some less. In small projects (around 10-50 person months of development) it seems to go well most of the time.

The problem starts with the big projects. Here you see great differences. You come to the daily meeting and you think you are in a different company.

 

In some projects (where it goes well) you see the project manager leading the meeting, setting priorities, making sure all are working together. In other projects the project manager couldn’t make it to the meeting so some other person from the project management team is facilitating the meeting (less of a lead and more of a facilitator). In these projects the various parties of the project take the agile spirit to the direction of the wild west, including show downs when the sun is high in the sky.

 

The down side of agile is that managers actually need to manage. It is a down side to some and an upside to others.

 

It is not a simple task moving from playing the commitment game to constantly navigating the ship through the stormy seas of software development. Some managers get a new spark in their eyes when introduced to agile, feeling revitalized with new zest and vigor. Some try to play the old game under the new dress and there things go awry and frustration follows.

 

When we start an implementation we ask the managers why do they want to go through this. Now we are starting to tell the managers what is the expectation from them. What will be the change in their day to day. We believe this will help us better align expectations with the managers and to make sure we have better results with the implementation.

Guest Post – Is starting with Kanban really easier than with Scrum?

Estimated Reading Time: 4 minutes

Today I’m proud to host a guest post by another AgileSparks coach – Yael Rabinovitz. Yael has been working with several clients on Scrum implementations and has recently started using the Kanban Method (I wonder who gave her that crazy idea…) and is sharing her thoughts about the first steps into both approaches. Without further ado, here’s Yael:

 

Is starting with Kanban really easier than with Scrum?

Kanban is often described as a way to achieve evolutionary change in an organization, creating less fear and resistance than more “revolutionary” approaches such as Scrum. Kanban indeed respects current processes. For example, in its first steps it does not impose role and responsibility changes, but encourages continuous small incremental and evolutionary improvements to your current system. But is it really an easier way to launch a change process? Always? The last few Agile implementations I was involved with that used Kanban as a starting point made me contemplate this topic. Here are some of my thoughts.

 

Implementing Kanban seems to be simple technically. Just visualize your current process, build the Kanban board and describe your process as a pull system. Kanban does not prescribe specific practices, but the team is expected to apply lean thinking tools such as limit the WIP, focus on flow, optimize the whole, stop the line and fix (focus on quality), find bottlenecks, protect the team from external interruptions, etc. This requires high maturity from the team. Unfortunately, in many cases, teams find themselves struggling at the starting point, especially when attempting to apply the “limit the WIP” step. This can be extremely difficult and requires good collaboration and shared responsibility between the development team
and the PO in order to allow balancing demand and
throughput. When this collaboration is not mature, teams tend to get stuck and find it hard to move forward, which leads to frustration and increases resistance to the change.

 

Shape the path – Script the way

Scrum’s prescriptive nature is more suitable for such immature teams as it “scripts the way” (from “Switch” by Chip Heath & Dan Heath) and provides best practices and definitions of the processes to be followed (daily standup, planning, demo, retrospective, clear definition of the PO/team responsibilities). Scrum provides a practical framework that makes the mindset shift easier, as teams can rely on the framework in this “SHU” phase (“Shu-Ha=Ri see: http://alistair.cockburn.us/Shu+Ha+Ri). The clear definition of the PO role vis-a-vis the team provides the needed collaboration and WIP is limited implicitly by defining the sprint’s backlog, which I believe teams find more natural to grasp.

Small work units

Another point to consider is the need to break features into small pieces or user stories. This is a huge challenge that teams usually struggle with from day 1 in both Kanban and Scrum. This is fundamental to improving the flow in the system, thus reducing cycle time in Kanban, and creates the building blocks of the product backlog and sprint backlog in Scrum. I find that Scrum guides you more safely and clearly in this task than Kanban, as the framework and practices for creating stories are explicitly built into the Scrum process. Questions like: who is the owner of the stories? When should the stories be ready? What is a good story? Who approves the results and when? The team has built–in answers when it uses Scrum as a starting point.

The energy for change

It is also very important to consider the “energy” that the organization needs to have in order to change. Agile practically requires the organization to “reinvent” itself and, in many cases, the beginning of the process is an excellent opportunity to do just that. At that time, the right levels of energy exist, with the “buzz” created by training and management’s attention, as well as the willingness and courage to make a dramatic move. Scrum suits this “let’s make a revolution” mindset much more than Kanban, which requires stamina, patience and long term thinking. Most teams don’t have the time and patience for ongoing improvement cycles – they want results now. They are willing to risk, speculate and go along with revolutionary ideas rather than patiently wait for evolution.

Early success

Being able to achieve small wins early in the process provides a lot of “back wind” to the process. Scrum’s sweeping nature allows identifying these small wins quicker, thus injecting positive energy into the process at a much needed stage. The adoption of cross functional teams and creation of a rhythm in the organization, with demos and retrospectives quickly gives a sense of progress. It is important to pick low hanging fruits and produce results quickly.

I found that in many cases, if a momentum for revolution exists both in management and teams, Scrum’s constraints are much better for keeping the teams focused and accountable. Once you reach a point where you have a backlog with well defined stories and you are mature enough to go to the next leap, it is time to implement Kanban in order to optimize the whole and focus on the flow of value across the system, improving your cycle time from start to end.

So, should we start with Kanban or Scrum?

The only right answer is: it depends. It depends on the organization’s state, the scale of necessary changes, the desired outcomes and the problems and pains that triggered the Agile initiative.

What I attempt to do in this post is challenge the widespread myth that Kanban is always the easiest way to start. I have found that, when management is on board with the right mindset, starting with Scrum can be safer and easier. The Retrospectives in Scrum provide the evolutionary mechanism Kanban advocates that allow the organization to continue and make small incremental changes and continuously improve beneficial aspects and weed out harmful attributes

Start with Kanban or Scrum but, since you are interested in this journey, it is important to start! Begin, keeping in mind why you started and what you are trying to achieve .We don’t want to waste our time with too much upfront planning anyway, after we begin the journey, the steps we should take to move forward will unfold.

“The Road goes ever on and on

Down from the door where it began.

Now far ahead the Road has gone,

And I must follow, if I can,

Pursuing it with eager feet,

Until it joins some larger way

Where many paths and errands meet.

And whither then? I cannot say.“…

 

Hobbits walking song, “The Fellowship of the Rings”,J.R.R Tolkien