Looking back at the 2004-2008 agile rollout at BMC, I am struck by how much was accomplished by a series of sensible decisions. At the time, it was a "simple" matter of the teams, my staff members, and me making sensible operational decisions. My knowledge as to why these decisions worked so well, however, would evolve at a much later time.

This Executive Update represents my current understanding of the deeper reasons behind the success of the BMC rollout. It reviews past decisions in light of knowledge, experience, and insights that evolved a long time after the decisions, for better or worse, had been executed. In general, it's about my making sense of things and sharing my insights with Cutter clients.

WHY AGILE?

The fundamental problem at BMC in 2004 was the deterioration of its PATROL/Performance Manager (now known as ProactiveNet Performance Manager).¹ The confluence of three painful factors made it a particularly tough problem:

1. Technical debt. Hardly any dollars were available for technical training during the 2000-2003 period and various folks began falling behind professionally. In addition, offshoring to India had been rushed to the extent that proficiency in software engineering was grossly neglected in the hiring process. Between these two factors, a high level of technical debt was the inevitable result.
2. Configuration management debt. The fundamental architectural concept of PATROL was easy to abuse from a configuration management standpoint. This vulnerability was compounded by all too many exceptions in the form of "just this special version for a single preferred customer." Over the years, the situation got out of control: too many coding branches; too few branches converging back to the mainline; and so forth.
3. Loss of heart. As the going was indeed tough, various execs at BMC concluded that other system management spaces presented more attractive opportunities. Hence, pressure mounted to shift the investment from PATROL toward products that were considered more promising.

Given these factors, we needed to develop what's known today as "escape velocity."² We clearly could not take the time to significantly reduce the technical debt that had accrued over a decade in millions and millions of lines of code. Nor could we effectively take head on the tendency to prepare special versions of the code to numerous customers -- the practice was too ingrained in the DNA of BMC. Last but not the least, the loss of heart amongst BMC executives needed to be turned around quickly. If we could not succeed, our budget would have faded away.

Agile offered an effective antidote to the "loss of heart" problem; if we did it well, we could demonstrate results quickly. While agile itself was not an antidote to neither technical debt nor configuration management debt, it could have bought up the time required to cope with these painful debts. Hence, going agile was a no-brainer.

THE SECRET SAUCE

The scale of the rollout was determined by organizational considerations; the business unit I was heading had more than 400 employees in seven countries. Since time was of the essence, we had to deploy at a fast pace. After a few difficult starts at the end of 2004, we developed the discipline required for launching a new team into agile just about every week. Figure 1 shows the progression of deployment in 2005 as recorded in the Rally system we were using.

Figure 1 -- BMC agile rollout in 2005.

Not much was known in 2004 about deploying agile at such a scale. In many ways, we had to roll on our own. We were aided by a few outstanding consultants and coaches, including Dean Leffingwell, Ryan Martens, (now) Cutter Senior Consultant Hubert Smits, and Jean Tabaka. Between experience gained by the teams, staff deliberations, consultant know-how, and my own convictions, we developed the "secret sauce" shown in Figure 2, whose four ingredients are defined below:

1. Leadership. It is primarily the ability of the executive in charge of the agile rollout to be intentional and to state clearly, loudly, and repeatedly that the end point is everyone in the business unit practicing agile. I was crystal clear in stating that not knowing the specific "speed bumps" we would encounter along the way was very different from knowing the end point (i.e., everyone doing agile) and being fully committed to reaching it.
2. Know-how. We did not think we needed to reinvent the wheel. Whenever and wherever possible, we availed ourselves to the expertise of various folks in the agile movement. (This ingredient is immensely more important today than it was back in 2004; progress here has been phenomenal.)
3. Flexibility. To quote the wise words of Cutter Senior Consultant Annie Shum, I believed in "context over content." The business consisted of three product lines. Incidentally, every product line optimized the agile implementation to its own needs.
4. Patience. My contention was, and still is, that an agile rollout is a three-dimensional change process: (1) you change the software; (2) you change the process; and (3) you change the organization. If you accept these premises, you must provide enough time for the teams and the organization to adjust to these simultaneous changes. Long-term perseverance is the most crucial factor in successful agile rollouts.

Figure 2 -- BMC’s secret sauce: 2004-2008.

At a certain point in time in 2009, I came to the realization that my own personal needs as a line executive could be quite different from those of the executives for whom I consult. In particular, I've found that numerous executives are reluctant to adopt the secret sauce without a clear handle on how they will govern the software process. Hence, I developed a training module on software governance and redefined the secret sauce to include five ingredients, as shown in Figure 3.

Figure 3 -- Cutter’s secret sauce: 2009.

WHAT WAS ACCOMPLISHED

Getting back to the BMC rollout, it was quite tricky at the beginning to assess how well we were doing. Hence, we reverted primarily to qualitative assessments. Here are just a couple of unsolicited accolades I received during the first couple of years of the rollout:

Working with the agile teams at BMC those last two years were some of the most rewarding in my career because of the sense of purpose, direction, and camaraderie that the process instilled in everyone. -- R&D architect

If we used waterfall on the BMC Performance Manager, we would still be in development. We would likely be cutting features right and left to try to bring the date back in. Changes requested along the way by the solutions teams would have been pushed back on rather than embraced. -- R&D director

After a couple of years of "agiling" day in and day out, I thought we were doing pretty well. However, I did not know that we were indeed doing pretty well. So I reached out to Cutter Senior Consultant Michael Mah who compared the BMC results against a sample of some 7,500 software projects in the Quantitative Software Management (QSM) database. The results, depicted in Figures 4 and 5, were beyond my wildest dreams.

Figure 4 -- Time to market as function of release size. (Source: QSM.)

Figure 5 -- Productivity index per industry sector. (Source: QSM.)

WHERE I FAILED

Successful as the agile transformation at BMC was, it completely failed in what I today consider the ultimate benchmark for a transformative rollout: it did not alter the company's philosophy and modus operandi beyond the level of "How do we make the sausage? We use agile methods."

I was blessed with two levels of management that trusted me to implement agile at BMC. Both Mary Smars to whom I reported and Dan Barnea who headed the whole of engineering at BMC were no dummies. They fully understood the risks associated with a massive transformation of the software process. In spite of the risks, they remained open-minded and consistently supportive even when things did not go too well.

Due to their whole-hearted backing, I was able to implement agile as a software method. From my perspective today, this means that I was primarily concerned with two strands: development and testing. I was able to "merge" these two elements so that testing could start before development was complete -- and testing informed development through tight feedback loops. The net effect at BMC (and later elsewhere) was faster/earlier delivery of products. Figure 6 depicts first-level agile implementation.

Figure 6 -- First-level agile implementation.

In my humble opinion, the very same agile principles hold at the strategic level. The only difference is that the interplay is not between development and testing but rather between strategy and delivery. One merges the two so that delivery can start before strategy is complete -- and delivery informs strategy through tight feedback loops. The net effect is faster/earlier delivery of products that satisfy the needs of the market. Figure 7 depicts second-level agile implementation.

Figure 7 -- Second-level agile implementation.

Sometime within the last couple of years, I met Russ Daniels, CTO of HP Enterprise Services. I was impressed to no end by his flexible thinking. In particular, I found the following words of his inspirational:

The real challenge, however, lies in how to go about solving problems when you don't understand them well enough to get to a viable solution.... When you don't have a clear enough understanding of the problem to create clear solutions, you have to iterate.

These words led me to think about third-level agile implementation (see Figure 8). At this level, one merges the problem and the solution so that the solution can start before the problem has been fully understood -- and the solution, incomplete that it might be, informs the problem through tight feedback loops. The net effect is faster/earlier learning.

Figure 8 -- Third-level agile implementation.

The BMC transformation was quite successful at the first level but did not really make it to the second level, let alone the third. While it might be a little unfair to characterize the BMC agile transformation in terms of a taxonomy that I evolved after the time of the BMC rollout, the inability to reach second-level agile implementation perplexes me to this very day. My hunch is that this probably reflects a lack of readiness at BMC at the time to accept unpredictability at the strategic level. To BMC executives, such risks outweighed the pros of a constantly evolving strategy in a way that is well correlated with the always-changing market needs. What I probably did not quite understand at the time was that BMC conceived strategy as largely fixed for prolonged periods of time. Rightly or wrongly, continuously grooming strategy (in a manner conceptually similar to the way one grooms the agile backlog) was perceived as too radical.

POSTSCRIPT

Looking back from a 2011 vantage point, two things are paramount in my mind with respect to the BMC agile transformation. First, upon leaving BMC to pursue an agile consulting career, one of my directors sent me an email that I cherish. Here is an excerpt:

The change you brought to BMC with agile is the single largest change to the development model that I have ever witnessed in my almost 20 years at BMC.

Second, my failure to progress agile at BMC from the first to the second level is the kind of failure from which I've learned an awful lot. The way I discuss, teach, and coach agile these days is anchored at my assimilating why I did not manage to accomplish more at BMC. Cutter clients who benefit today from my later wisdom ultimately owe it to the 2004-2008 rollout of agile at BMC.

ENDNOTES

¹ For details on ProactiveNet Performance Manager (PATROL), see www.servicetech.com.hk/bmc/patrol.html.

² Moore, Geoffrey A. Escape Velocity: Free Your Company's Future from the Pull of the Past. HarperBusiness, 2011.