"There is no doubt that agile is much more suited for innovative companies than traditional waterfall methods are. But is agility the natural path to innovation?"

An Obstacle to Innovation

The key practices of agile development -- timeboxed iterations, prioritized features, frequent customer feedback -- are about risk reduction, not innovation. Creativity and true innovation are driven out by agile's relentless focus on working-software-delivered-on-time.

The Midwife of Innovation

Innovation is the result of collaboration instead of following a detailed master plan, and that's what agile development is all about. By emphasizing face-to-face collaboration, individual creativity, and rapid feedback, agile helps organizations give birth to innovation.

Opening Statement

A few months ago, I started introducing agile development to a highly innovative company. They are world-market leaders in their domain, and it's basically their technical expertise they hold responsible for their position. As I usually do, I started by teaching them agile planning and estimation techniques, and they seemed to be quite happy with them -- except the lead for the team that was responsible for the innovative core of the system. "This looks quite good," he said as he approached me after the first planning session, "but where is the time we need for research? You see, we are 'research and development,' and all we were talking about just now was 'development.' " I answered something along the lines that their velocity still leaves enough time for unplanned research, but he seemed to be more satisfied with my answer than I was. Agile development should point the way out of the process problem, but as I was explaining it, it seemed to me more like tweaking the planning process.

When I started with agility 10 years ago, I believed that agility naturally led to innovation -- that agile development was the obvious way to write innovative software. In their worthwhile book Artful Making: What Managers Need to Know About How Artists Work [1], Cutter Fellow Rob Austin and Cutter Senior Consultant Lee Devin argue that industry could learn from the disciplined way theater actors and other "creative professionals" deliver innovation repeatedly and reliably on time. One of their examples of where these lessons already have been applied to industry is agile software development. And now I had to show my client a way out of agility so they could be innovative? I must be missing something in this picture!

There is no doubt that agile is much more suited for innovative companies than traditional waterfall methods are. But is agility the natural path to innovation? If you look at the principles behind the Agile Manifesto, ¹ the first one reads: "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software." This is about delivering something some customer wants, needs, and is willing to pay for. This is about finding and fulfilling requirements. This is not about inventing cool things no one else has ever thought about before.

It's true that agile uses a lot of techniques for creativity, such as direct face-to-face communication, iterative work, and fast feedback, to name just the most important ones. And it promotes the idea that software development is an act of invention and development, rather than an act of production, thus making it an innovative act. But it also puts IT development under the close control of the business, with planning workshops, backlogs, prioritization, product owners, and so on. Where is the space for research? Where is the space for innovation?

One component of my quest to solve this problem is this issue of Cutter IT Journal. The good news is that we managed to assemble a whole spectrum of answers. The bad -- or maybe not so bad -- news is that some of these answers contradict each other, so instead of an ultimate truth, you will find an interesting debate. We hope the resulting insights will be helpful to your organization.

AGILE IS AS AGILE DOES

Our first author, Erik Stein, opens his article "Innovation: Agile with Intent" with the following battle cry: "In and of themselves, agile practices do not foster innovation. In fact, left untended in product development situations, they are quite likely to become obstacles to innovation." He supports this bold pronouncement with the observation that innovation is merely a matter of the surrounding business environment, rather than of some software development practices. If you use agile methods to reduce the project risks and to control the development team, innovation is not really likely to happen. "Timeboxing is the antithesis of creative thinking," Stein declares. But he also sees exculpatory aspects: "Agile practices can make an organization more adaptive and help management maintain an innovative culture." He suggests that you distinguish between production-like fulfillment of -- possibly innovative -- business requirements and development of innovative software. If innovative software is your goal, enable the team by agile practices, but don't put them under the rigid constraint to deliver value-added software with each increment. After all, innovation is about trial and error. That said, Stein does recommend defining checkpoints where project results can be reviewed and decisions made as to whether to continue a project or not.

USER-UNFRIENDLY AGILITY

Next, Carol Barnum, director of the Usability Center at Southern Polytechnic State University (USA), describes an interesting case where the introduction of agile development decreased a client's ability to maintain the quality of their innovation. When transforming their process from a waterfall methodology to Scrum, the client also jettisoned the "formal" usability testing done by the Usability Center. In its place, they installed a proxy customer who also had to decide about usability issues. This resulted in a decrease in usability and, therefore, a decrease in customer satisfaction.

I think there are two lessons you can draw from this case. First, when introducing agile into a successful organization, don't lose your ancestry. Do a careful assessment of what made you successful up to now and which practices supported your success. This is especially important in an innovative environment. In Barnum's case, usability was part of the innovative power of her client. Instead of force-fitting usability testing into the role of an agile customer proxy, this organization would have done better to fit their practice of usability testing into their agile behavior. "Usability testing must be agile," Barnum asserts. This may be one of the examples where a "shrink-wrapped" agile process backfired, and a carefully customized one would have been better.

The second lesson is that the customer is still the most difficult role in agile development. Project success depends on the customer's innovative and creative power more than on anything else in the project.

THE BEST-LAID PLANS ...

Our third article features a case study about a failed innovation at "BestSpy," a maker of intelligence management software. Cutter Senior Consultant Sam Bayer tells the story of an agile team that had set off to conquer a new market for their company with a new downsized version of BestSpy's existing product. The team managed to get upper management support and went through several successful iterations, receiving highly positive appraisals from their prospective customers. However, before launch, they were stopped because the sales department was under so much pressure from their established market that they could not afford to launch the new product, even though its features had been well received in the new target market. Launching the new product would have meant reducing attention to some current and prospective customers of the flagship product.

I think this story offers excellent support for Stein's claim that it is management that makes an organization innovative -- or doesn't, as in Bayer's tale. "Agile is the midwife of innovation," Bayer claims. "It can't guarantee the timing, health, or nature of the baby, but it plays a major role in its delivery."

SHOW ME THE MONEY: AGILITY FROM A FINANCIAL PERSPECTIVE

Responsible innovation requires organizations to closely manage the risks associated with the new territory they hope their innovation(s) will conquer. In "Return on Agility: Financial Perspectives on Agile Development," Jeffrey Wu and Duff Bailey draw an interesting parallel to managing the risks of innovative IT projects using agile methodologies. In their article, Wu and Bailey compare agile development to the techniques used in an industry that is nearly completely based upon managing risks -- the financial services industry. In three case studies involving financial firms, they show how IT project managers drew "put options" and "call options" to reduce the risk of innovation and to maximize the gain from future opportunities. Wu and Bailey's metaphor offers a new view on how to explain agility to businesspeople who are "not interested in technical details." As the authors observe, "The paradox is that, outside of technology, most companies already employ agility as they pursue a business strategy through adaptive tactics." It is for this reason, the authors conclude, that "while agile development proponents may face cultural challenges within IT, especially with regard to infrastructure, they may find unexpected allies among their business sponsors."

AGILE, OPEN SOURCE, AND INNOVATION: A GOLDEN TRIANGLE?

In our final article, Jon Avotins, Florin Dragu, Alan Skorkin, and Adrian Cretu-Barbul draw a connection between using open source software and agile development to build an innovative environment. The open source community provides some of the most innovative ideas and tools in the computer industry. To fully exploit the innovative potential of open source, they argue, your own organization has to be compatible with the dynamic nature of open source communities. And this is where Avotins and his coauthors make the connection to agile development: "Agile processes can help you utilize your open source tools/libraries more effectively and engage the community more fully. More importantly, using open source with an agile process in place can promote just the right environment to allow innovation to flourish within an organization." At first reading, this contradicts Stein's opening statement that agile is not the key to innovation. But on second glance, it supports Stein's statement that agile may help breed innovation if used in the right environment. The authors conclude with a set of strategies for organizational innovation that may help to close the triangle of agile, open source, and innovation.

FITTING AGILE TO WHAT WORKS WELL

Now, where do these discussions and case studies leave us with regard to my initial question of how agile development is connected to innovation? I think the bottom line is that agility may help you to embed software development into an innovative organization. However, if the software is the innovation, you have to make sure that you customize agility to support the core elements of your innovation. This applies both to the practices aimed at supporting the business focus of development and to special practices in your organization that contribute to your innovation. Don't stop activities that are working well just because "our agile process does not support this." Rather, make sure you understand the contribution of these activities and try to fit them into the agile way of working. Understanding values and principles is more important here than just following a book. Supporting an innovative organization with agile software development requires experience and knowledge. In the case of my highly innovative client, we're still balancing business focus and innovation. Neither of them can live without the other.

NOTES

¹See http://agilemanifesto.org/principles.html.

REFERENCE

1. Austin, Rob, and Lee Devin. Artful Making: What Managers Need to Know About How Artists Work. Financial Times/Prentice Hall, 2003.

ABOUT THE AUTHOR