You are currently browsing the monthly archive for December 2009.

I came across a great article over at the ACM blog regarding how Agile Facebook is and the kind of value they are seeing from using these kinds of processes even though they are well out of startup mode.   A lot of people think that Lean-derived techniques such as continuous deployment, Scrum, and Agile are really only for small, startup-type companies, but just as Lean manufacturing isn’t just for small workshops, Lean software development can deliver astounding results to any company – especially large ones that have become complacent and allowed a lot of non-value producing process to creep into their methodology.


Perhaps the most interesting and revealing aspect of Robert’s talk was the discussion of Facebook’s somewhat unique development process.  At the surface it appears to have the contradictory goals of: minimizing down time, scaling fast, and extremely quick code updates.  First, Facebook developers are encouraged to push code often and quickly.  Pushes are never delayed and applied directly to parts of the infrastructure.  The idea is to quickly find issues and their impacts on the rest of system and surely fixing any bugs that would result from these frequent small changes.

Second, there is limited QA (quality assurance) teams at Facebook but lots of peer review of code.  Since the Facebook engineering team is relatively small, all team members are in frequent communications.  The team uses various staging and deployment tools as well as strategies such as A/B testing, and gradual targeted geographic launches.  This has resulted in a site that has experienced, according to Robert, less than 3 hours of down time in the past three years.

In Eric Ries’ latest post over at Startup Lessons Learned, he addresses many of the most common objections he’s heard regarding continuous deployment.  According to Eric, most of the objections boil down to the following two points:

      1. That mission critical customers won’t accept new releases on a continuous basis.
      2. That continuous deployment leads to lower quality software than software built in large batches.

I think he addresses both of these points extremely well and goes further in illustrating the improvements that continuous deployment brings to an organization:

  1. Faster (and better) feedback. 
  2. More automation. 
  3. Monitoring of real-world metrics.
  4. Better handling of intermittent bugs. 
  5. Smaller batches

Be sure not to miss out on the great discussions in the comments thread as well.   One comment Eric made that really stood out to me was his mention that the dichotomy between “fewer, more stable releases” and “faster, less stable releases” is a false one and that what continuous deployment truly brings to an organization is faster and more stable releases.  

This reminded me of a lot of objections I’ve heard regarding Lean in that doing the right thing up front (building infrastructure, identifying and planning around your constraints, etc.) will give you better quality but slower development; in fact the truth (as borne out both in manufacturing and software development) that doing the right thing up front may cost more initially but you’ll end up letting people work faster while also increasing quality and customer value over the life of the project.

2009 was a very good year for me, and I hope it was for all of you.  Now that it’s mostly over, I feel like much of what I accomplished (and the way I’ve responded to change) this year was all was oriented around setting myself up for great things next year:

  • Although I was laid off from my publishing producer role as part of the big restructuring and refocus at EA, I was quickly able to join a new company that is in the midst of a Lean/Agile revolution as a Release Manager.  The role reports directly to the CTO and has a strong mandate to deliver results to an established company looking to reinvent itself in order to compete against highly agile startups moving into their market.  It’s a challenging post, and I think the idea of “Go Big or Go Home” really applies here.   The best part is I’ll be able to share my lessons learned on this blog.
  • I’m back in Los Angeles now, and I have to say that now that I’m back from the Valley I’m realizing how much I missed this place.  The tech industry is a bit smaller here, but you really get to know people at all the companies here and it feels much more like a community.  I’m hoping to leverage some of my connections here into some quality interviews and guest writer posts here on this blog.
  • I’ve gotten to know some of my Lean guru friends better, as well as met some new ones.  Their insight and influence will be a major part of the way this blog takes shape over the next year.

Looking ahead to 2010 I’m excited to see how everything that has set itself up in 2009 plays out.   Definitely look for more posts on this blog now that I’m doing real configuration management again at a Lean company!

All the best for 2010,


Jason Lenny is a Lean manager and innovator with over ten years of experience managing production software development pipelines and process.