Once again TED delivers with another amazing talk – this time, it’s on the concept of flow, which should be a familiar idea to everyone who works with Lean.  Interestingly, however, what Mihaly is addressing here is the role of psychological flow and its impact on our daily lives.

Mihaly Czikszentmihalyi asks, “What makes a life worth living?” Noting that money cannot make us happy, he looks to those who find pleasure and lasting satisfaction in activities that bring about a state of “flow.”

If you haven’t seen it yet, Timothy Fritz’s post “Doing the impossible fifty times a day” on exactly how they are doing continuous deployment at IMVU is very fascinating.

“Our tests suite takes nine minutes to run (distributed across 30-40 machines). Our code pushes take another six minutes. Since these two steps are pipelined that means at peak we’re pushing a new revision of the code to the website every nine minutes. That’s 6 deploys an hour. Even at that pace we’re often batching multiple commits into a single test/push cycle. On average we deploy new code fifty times a day.

So what magic happens in our test suite that allows us to skip having a manual Quality Assurance step in our deploy process? The magic is in the scope, scale and thoroughness. It’s a thousand test files and counting. 4.4 machine hours of automated tests to be exact. Over an hour of these tests are instances of Internet Explorer automatically clicking through use cases and asserting on behaviour, thanks to Selenium. The rest of the time is spent running unit tests that poke at classes and functions and running functional tests that make web requests and assert on results.”

Be sure to check out the actual article for even more great detail, particularly around how they actually do their deployments once their automated test suite has assured a high level of quality.

I came across a great article over at the ACM blog regarding how Agile Facebook is and the kind of value they are seeing from using these kinds of processes even though they are well out of startup mode.   A lot of people think that Lean-derived techniques such as continuous deployment, Scrum, and Agile are really only for small, startup-type companies, but just as Lean manufacturing isn’t just for small workshops, Lean software development can deliver astounding results to any company – especially large ones that have become complacent and allowed a lot of non-value producing process to creep into their methodology.


Perhaps the most interesting and revealing aspect of Robert’s talk was the discussion of Facebook’s somewhat unique development process.  At the surface it appears to have the contradictory goals of: minimizing down time, scaling fast, and extremely quick code updates.  First, Facebook developers are encouraged to push code often and quickly.  Pushes are never delayed and applied directly to parts of the infrastructure.  The idea is to quickly find issues and their impacts on the rest of system and surely fixing any bugs that would result from these frequent small changes.

Second, there is limited QA (quality assurance) teams at Facebook but lots of peer review of code.  Since the Facebook engineering team is relatively small, all team members are in frequent communications.  The team uses various staging and deployment tools as well as strategies such as A/B testing, and gradual targeted geographic launches.  This has resulted in a site that has experienced, according to Robert, less than 3 hours of down time in the past three years.

In Eric Ries’ latest post over at Startup Lessons Learned, he addresses many of the most common objections he’s heard regarding continuous deployment.  According to Eric, most of the objections boil down to the following two points:

      1. That mission critical customers won’t accept new releases on a continuous basis.
      2. That continuous deployment leads to lower quality software than software built in large batches.

I think he addresses both of these points extremely well and goes further in illustrating the improvements that continuous deployment brings to an organization:

  1. Faster (and better) feedback. 
  2. More automation. 
  3. Monitoring of real-world metrics.
  4. Better handling of intermittent bugs. 
  5. Smaller batches

Be sure not to miss out on the great discussions in the comments thread as well.   One comment Eric made that really stood out to me was his mention that the dichotomy between “fewer, more stable releases” and “faster, less stable releases” is a false one and that what continuous deployment truly brings to an organization is faster and more stable releases.  

This reminded me of a lot of objections I’ve heard regarding Lean in that doing the right thing up front (building infrastructure, identifying and planning around your constraints, etc.) will give you better quality but slower development; in fact the truth (as borne out both in manufacturing and software development) that doing the right thing up front may cost more initially but you’ll end up letting people work faster while also increasing quality and customer value over the life of the project.

2009 was a very good year for me, and I hope it was for all of you.  Now that it’s mostly over, I feel like much of what I accomplished (and the way I’ve responded to change) this year was all was oriented around setting myself up for great things next year:

  • Although I was laid off from my publishing producer role as part of the big restructuring and refocus at EA, I was quickly able to join a new company that is in the midst of a Lean/Agile revolution as a Release Manager.  The role reports directly to the CTO and has a strong mandate to deliver results to an established company looking to reinvent itself in order to compete against highly agile startups moving into their market.  It’s a challenging post, and I think the idea of “Go Big or Go Home” really applies here.   The best part is I’ll be able to share my lessons learned on this blog.
  • I’m back in Los Angeles now, and I have to say that now that I’m back from the Valley I’m realizing how much I missed this place.  The tech industry is a bit smaller here, but you really get to know people at all the companies here and it feels much more like a community.  I’m hoping to leverage some of my connections here into some quality interviews and guest writer posts here on this blog.
  • I’ve gotten to know some of my Lean guru friends better, as well as met some new ones.  Their insight and influence will be a major part of the way this blog takes shape over the next year.

Looking ahead to 2010 I’m excited to see how everything that has set itself up in 2009 plays out.   Definitely look for more posts on this blog now that I’m doing real configuration management again at a Lean company!

All the best for 2010,

One of the constraints that makes moving towards continuous deployment difficult for some teams is the downtime required with web app deployments.   Managed Chaos just posted an article on how to update web applications without destroying in-progress user sessions.   In this specific example he illustrates how to do this using a Tomcat cluster, but the fundamental principles apply to any tiered web application architecture.

Databases are, of course, a bit trickier since you’ve potentially got lots of dependency problems; luckily, the Exortech blog recently addressed this issue as well.

The more I’ve been thinking about continuous deployment the more I feel like it’s really going to be a game changer for companies that adopt it.  The barrier for entry is relatively low (and can be well understood before embarking on a project to adopt the process) and the reward of being able to reduce batch sizes on merges and (more importantly) give your product owners the ability to quickly get features in front of customers is key for any business.

I came across this flowchart that helps you determine the best chart to use based on the kind of data you are trying to get across.   I know that for me I have a tendency to rely on a few “go-to” charts that I’ve come to rely on when in fact there may be a better option out there that more elegantly expresses exactly what I’m trying to communicate.  This chart looks like a great resource to use to check your instinct against best practice.

Click on the image for a larger and easier to read version.

Source: http://extremepresentation.typepad.com/blog/2006/09/choosing_a_good.html

Jakob Ehn over at GeeksWithBlogs has written up a great list at http://geekswithblogs.net/jakob/archive/2009/05/23/tfs-team-build-2010-whatrsquos-new.aspx.   I’m particularly excited about the possibility of using templates for build definitions (I remember many an hour spent duplicating an existing build definition “the long way” with TFS 2008.)  It’s very cool that Microsoft decided to add gated check-ins as a core feature; we were able to get it working in the previous version, but it required a lot of creative engineering to get it working correctly.

I’ve just finished getting my local VM TFS sandbox up and running, and I have to say I’m very impressed with the new installation and configuration process.  There still seem to be the usual arcane permissions magic that needs to be done in order to get everything playing together nicely, but nothing that you can’t figure out with a bit of trial and error and/or making service accounts domain administrators. 🙂

Craig Tadlock (over at the Tadlock Enterprises Blog) made a couple good posts recently about Team Foundation Server and Team Build 2010:

Work Item Classifications
Work Item Relationships

He also came across this great blog that is focused on doing automated testing:


as well as this link discussing custom workflow items:


I’m setting up a local TFS test environment for myself now and hopefully soon should have some interesting information on setting up Lean build systems using this great platform.  I was lucky enough to work at a company that used the previous incarnation of Team Foundation Server and I have to admit I’m pretty excited to see what goodies Microsoft is giving us on this go-around.

Eric Ries has posted another fascinating article on continuous deployment, which is one of his more controversial (and brilliant, in my opinion) proposals.

To me, continuous deployment seems like a great idea for any start-up or environment where the value of fast iteration is more important than absolute stability (although stability is quite creatively addressed via the use of split testing in the production environment.)

Check out the article on why CD is worth using at http://startuplessonslearned.blogspot.com/2009/06/why-continuous-deployment.html and decide for yourself; unfortunately I’m not in a place right now where I can put this into practice, but as soon as I get back to a start-up or an immature product I’ll definitely be lobbying hard to put continuous deployment in place.  If any readers get a chance to put it into practice I’d love to hear your stories.


Jason Lenny is a Lean manager and innovator with over ten years of experience managing production software development pipelines and process.