Value Stream Mapping the CM Pipeline

In this article we’re going to dig into using value stream mapping to find inefficiencies in our build and release pipelines.   A value stream map is a Lean method for visualizing and understanding your processes and is a great starting point to finding and weeding out inefficiences.

How to create a value stream map

The simplified form of value stream maps which I’ll be showing here are actually pretty straightforward to create:

• Start with the entry point into your process and generate a workflow that ends up with value created for your customers.   Be sure that each of your workflow steps are real work – someone should be actively doing something for each step (waiting for a build, or really any kind of waiting, is not a work step in a value stream because nothing is being produced.)  
• Next, notate the average time spent doing each step as well as the average time spent in between each step.  Estimates are OK, but if you actually do the experiment and collect some real world data you will probably find some pretty surprising numbers.  If you’re using a full-fledged workflow tool you may be able to get these timings out of your reporting system, but if you’re not tracking this in software then the best way I’ve found to get this data is to use Excel to track all the timings for one or two instances of the process per day for a week or two.  After you’re finished you can average your timings to come up with reasonably accurate estimates.   Be sure to also count the number of times you rework issues and what rework steps are associated with doing so.
• Finally, go back and notate any rework loops and any other ways to exit the process aside from creating value (e.g., cancelling a feature would be an early exit to a feature development process that does not create value.) Possible flows where no value gets created are huge potential time sinks, especially in cases where you rework an issue or feature many, many times only to leave it on the cutting room floor.

A real world example

Below is an example value stream map I’ve created for a typical build pipeline.  You’ll notice that I’ve mapped an entire defect resolution process from discovery through resolution, and that’s because it’s very important to always look at your systems within their broader context; the CM pipeline doesn’t exist within a vacuum and looking at your systems holistically is really the best way to approach solving systems level problems.

1. After 3 hours of build testing a defect is found by QA.
2. An hour goes by waiting as the QA team discusses the issue internally and verifies it is reproducible.
3. QA spends an hour writing up the official bug report which includes all information needed to fix the issue.
4. Sixteen hours go by (on average) as we wait for the next change control board meeting where the bug is assigned to a developer.
5. The developer spends two hours making the fix and checking it in (this includes any unit testing or other pre check-in requirements.)
6. An hour goes by as the continuous integration system monitors for and pools check-ins.
7. The build runs for an hour.
8. QA only tests one build per day, so we wait one day for the fix to be ready for deployment.
9. The deployment runs for an hour.
10. After about a half day QA gets to the test in question.
11. QA spends three hours testing.
12. QA spends an hour updating the (hopefully) fixed issue with updated information and adding it to their regression suite.  This is where the value is created in this example – the end product has one less critical defect.

For this example rework happens on average once on four out of five bugs submitted.

The value stream map

Mapped out using Visio, the value stream map looks like this:

Metrics

Once you’ve got your value stream mapped out you can find some interesting overall metrics. These are great to keep track of so over time so that you can see how you’re tracking at the macro level, especially since within configuration management it’s easy to get lost in the details.

You can find an overall ideal efficiency by dividing the number of hours spent doing real work by the total number of hours in the entire process, without including reworks.  In this value stream map there are 13 hours of real work happening (for the ideal situation where the defect is fixed on the first attempt) and 43 total hours of cycle time total.   This gives us an ideal efficiency of roughly 30%.  A good target I’ve found to shoot for with build systems is around 20-25% – you never want to get too close to 100% because at that point any new work added to the system will cause thrashing, and work coming into the build pipeline is spiky even on the best of days.  

Another interesting value you can pull out of this is rework cost, which can be found by adding up the total number of hours spent on each rework cycle.  In this example we add 21 total hours each time we come to a fix failed state.  Since we found we average one rework cycle on four out of five bugs our rework average is 80%.  Rework metrics are great resources to get a feel for quality coming into the system and amount of waste being produced based on quality issues.  

Next steps

One of the great wastes that value stream maps can highlight is time spent in rework cycles.   Because in our example we lose 21 hours per rework cycle (and we’re sitting at an 80% rework average) anything we can do to bring that iteration time or rework average down will be a huge win.  If we can increase development time, for example, in order to our reduce rework average then we’ve made a signficant overall improvement through making a change that in isolation would have appeared to be counterproductive.  Keep in mind that any improvements made during steps in a rework cycle have the potential to be gained every time that rework cycle happens, so if you’ve got a high rework average then spending time reducing rework cost is going to give you a lot of bang for your buck, but if your rework average is very low you may be able to exchange a higher rework cost for a lower ideal efficiency.

A great place to start to look for optimizations is by analyzing your “waiting steps.”  These are pure waste (they are idle inventory in Lean terms) and are prime targets for efficiency gains from simply reducing the time spent there.  In our example the most egregious waiting periods are the ones where we are spending two days for the original defect to be scheduled for work, a day for QA to pick up the fix in the next deployment, and four hours for a QA engineer to get a chance to take a look at the bug in the test environment.  Because there’s no real work happening here it’s often easy and cheap to find ways to reduce these waiting periods.

Process steps are also good potential candidates for making improvements, but here things are more complicated because someone is actually doing some kind of productive work and you’ll need to do some investigation into the details to find out what’s happening there.  In this example there aren’t any process steps that jump out as particularly slow, but if our build and deployment times started creeping up it would definitely be worth spending some engineering resources to bring that back down to a level that makes more sense.

Going further, the next steps would be to try to think about your map overall:  

• What other kinds of timing improvements can you find?  
• What other teams could you bring on board to improve iteration speed?  
• Do you really need to do every step?  
• Does this workflow even make sense now that you’ve written it out?  
• Is there  a way to reduce efficiency in one area that increase overall effiency?  
• Are there any more hidden rework cycles in here that can be looked at?  
• Are there any possible flows where value is never created, and if so, how can these be avoided?  

Summary

Taking the time to analyze your processes using value stream maps can be a great way to look at your existing problems in a new way.  I hope that I’ve given you some ideas on ways to get started looking at your systems in terms of a value stream maps and that they can be as useful for you as they have been for me.

If you find that creating your value stream creates more questions than answers then you’re absolutely doing it right.  Value streams are a great method for highlighting waste and inefficiencies, but they aren’t a method for determining how to fix the problems.  In order to find the way to fix any problems you find you’re going to need to do some root cause analysis, which is a subject I hope to get into in more detail in a future article.

Advertisement