"The Agile and Lean movements clearly have a lot of life left in them. My expectation is that we’ll spend the next decade or more adopting disciplined Agile strategies to enable us to solve more complex problems and make possible a truly Agile enterprise."

Opening Statement

I'm honored to be the Guest Editor of this month's Cutter IT Journal. This issue describes the experiences and findings of senior consultants and researchers who have been actively involved with helping organizations around the world apply Agile strategies in enterprise environments. The term "Agile software development" was first coined in 2001, and since then Agile techniques and ideas have been applied in all industries, all geographies, and, most importantly, in a wide range of situations. But many of these successes are small-scale within organizations -- a few projects here and there have benefited, but rarely do you hear about large-scale enterprises successfully making a complete transition to Agile. The implication is that we still have a long way to go.

There are several important themes running through these articles. First, to succeed at Agile, you need to take a hybrid approach composed of strategies from Scrum, XP, Agile Modeling, Kanban, Unified Process, and, in some cases, even traditional methodologies. Second, when organizations apply Agile techniques at scale, they need to adopt more robust -- or dare I say "mature"? -- strategies than the Scrum community typically promotes. Third, Agile teams in today's enterprises need to go beyond their team-focused strategies and become enterprise-aware. Fourth, to transition to true enterprise agility, you need to adopt greater discipline at the practitioner, team, and even organization level that goes beyond the mainstream Agile philosophies that emerged in the 1990s.

In our first article, Ipek Ozkaya, Robert L. Nord, Stephany Bellomo, and Heidi Brayer of the Software Engineering Institute set the pace by summarizing results of research into the actual practices of Agile teams. This study explored how organizations are adopting hybrid approaches that build on the strengths of Scrum and XP with Agile architecture practices to achieve the benefits of Agile across the lifecycle. They share evidence of this hybrid strategy gathered via interviews with representatives of five public- and private-sector organizations that operate in highly regulated markets. They found that when experienced practitioners encountered scaling problems, most did not apply Agile practices in isolation: "Rather, they used their expertise to creatively combine them with other practices, especially architecture, to respond effectively to stability issues while rapidly fielding projects." The architecture strategies these practitioners used include initial architectural modeling/envisioning, monitoring technical debt, and addressing nonfunctional requirements. The authors conclude, "The experiences of these organizations support the stance that a more expansive application of hybrid practices is not only necessary but essential in balancing the opposing objectives of speed and stability."

Cutter Senior Consultant Bhuvan Unhelkar, author of The Art of Agile Practice, provides a complementary view in his article on the Composite Agile Method and Strategy (CAMS) and Agile business. Unhelkar highlights the "need to consider the value Agile offers to business, far beyond that originally anticipated by the proponents of Agile in the software domain." His CAMS framework shows how "embedding agility within formal planning results in an overall balance that not only benefits close-knit software development teams, but also the entire business." He argues that CAMS promotes a balanced strategy that enables the benefits of Agile at an organizational level. I believe you will find that CAMS presents a well-considered, practical adoption and deployment strategy for Agile.

The CAMS framework encompasses three levels: the development level (typical of Agile methods such as Scrum and XP), the project level (characterized by a composite/hybrid Agile approach), and an organizational level. Unkelkar argues that a primary Agile challenge is applying Agile across an organization. He notes, "What seems to work for a small, close-knit team suddenly seems 'all over the place' when applied across the organization." It is here that the concepts of the Disciplined Agile Delivery (DAD) process framework can offer assistance. As Unhelkar observes, "The DAD framework provides a carefully construed mechanism that not only streamlines IT work, but more importantly, enables scaling." The article further highlights the need to avoid what the author refers to as "method friction." When several different methods -- each introduced by a different role -- are in use within the organization, it can cause friction, and in turn reduce quality and flexibility. Such DAD concepts as a focus on solutions (not just software) and on scalability can enable the composite Agile approach CAMS represents.

In our third article, Al Shalloway argues that Agile success has eluded many larger IT organizations and software development organizations. "Rather than productivity and profit," Shalloway says, "they experience mediocre results or worse." He believes that the Agile movement's failure to adopt a solid systems perspective at the enterprise level is a significant factor. He writes, "It is not enough to try to build up team by team, with management trying to remove impediments along the way; instead, it requires an integration of business stakeholders, middle management, and effective teams. It requires both systems thinking and the discipline that systems thinking demands."

Shalloway suggests that bringing discipline and coordination to and across Agile teams requires going beyond classic Agile with its team-centric point of view. In particular, "it requires extending Agile with Lean principles and practices," a mindset he calls "Lean-Agile." Chief among these is systems thinking, which encourages organizations to examine the development workflow as a whole, seeing how each part affects the others. He notes that "local optimizations without this view may have little -- or even a harmful -- impact on the whole." For example, the "self-organizing team" is a very good Agile practice. It works at the team level because teams can use their own local knowledge to adjust to their particular situation. At scale, however, a broader perspective is required. "Management provides this perspective," Shalloway writes, "by creating the context within which teams work together. When several teams are involved, a lack of management oversight inevitably leads to a lack of team coordination, which leads to wasted effort."

This Lean-Agile mindset suggests three new disciplines for Agile organizations. The first is that stakeholders cannot begin more projects than the development organization has capacity for. In other words, manage your work in progress across your overall portfolio. Second, project teams must work in unison. If each team chooses stories from its backlog at its own discretion, the completed features will likely fail to produce end-to-end functionality. Third, Agile teams must let someone who sees the bigger picture decide what they should be working on. This implies the need both for a Lean-Agile enterprise architecture effort as well as a governance effort. Shalloway ends with the observation that "Agile methods informed by Lean Thinking lead to the disciplined type of Agile required for the Agile enterprise. We must remember that Agile is not about team-iterative development, but rather about the incremental delivery of business value."

Next, IBM's Matt Ganis and Sanjeev Kumar V Marimekala describe several key practices for becoming more disciplined, and thereby more effective, in your approach to Agile solution delivery. They describe how they used the DAD framework to create a process within their organization that supports a great number of both Agile and traditional project deployments. The authors write:

Early on, we realized that many of our enterprise processes wouldn't be eliminated with the advent of an Agile approach. Many (if not all) of our projects, while claiming to be Agile, needed to collaborate with other teams/resources such as IT deployment (data center teams), architecture teams, database administrators, and others. While our Agile teams were creating usable software at a steady drumbeat, we struggled with the inability (or lack) of IT staff to deploy applications to the network where they could begin realizing value.

Ganis and Marimekala recommend the adoption of six practices (or "disciplines," using Shalloway's vernacular) to facilitate application deployment. First, promote open communication channels across your IT organization to enable people on disparate teams to work together more effectively. Second, focus on reducing waste (or better yet, eliminate it completely) in order to streamline your overall strategy. Third, strive for the best utilization of resources from a business perspective and thereby maximize the return on your IT investment. Fourth, apply resources where needed so as to effectively utilize team skills. Fifth, maintain a proper lead time for each project. In other words, invest in project inception/initialization activities such as initial requirements envisioning, architecture modeling, planning, and team building. Sixth, keep it simple -- an echo of the ninth principle behind the Agile Manifesto.

We round out this issue of Cutter IT Journal with an article by South African software industry entrepreneur Themi Themistocleous, who explores the softer aspects of Agile software delivery. As Themistocleous observes:

With Agile development, it becomes even more important to have the right people in the right roles, as all team members are required to be self-organizing and cross-functional and to take responsibility for the success of the project. Personality types are starting to be recognized as an important factor in the success of Agile projects, and the first studies and books on the topic are making their way into the Agile world.

Themistocleous describes three important factors that contribute to software development success. First is the culture of the organization. In particular, he describes how the best organization he ever worked for strived to provide an egalitarian, nurturing environment that enabled people to work well together. The second factor is the correct use of development processes. For example, he describes how the DAD framework provides ways to provide effective governance without needless bureaucracy by focusing on process goals instead of on paperwork. The third factor is enabling people to take on the roles best suited for them.

I want to leave you with several important observations. First, organizations are finding that it's a lot of hard work figuring out how to make all this Agile stuff actually work in practice. Although the Scrum message of "start with a small kernel and add what you need" is attractive, in practice it's proving difficult, time-consuming, and expensive. As you will see in these articles, it is likely more effective to start with a framework such as Disciplined Agile Delivery, which describes a hybrid approach that supports a full delivery lifecycle. Furthermore, DAD promotes a solution focus, not just a software focus, and a strategy that is enterprise aware and not just team aware. More importantly, it answers the question "How do all the Agile practices and strategies fit together?" with its goal-based approach. Instead of starting with something small, why not start with something sufficient?

Second, the Agile and Lean movements clearly have a lot of life left in them. My expectation is that we'll spend the next decade or more adopting disciplined Agile strategies to enable us to solve more complex problems and make possible a truly Agile enterprise. Stay tuned for future writings on these subjects!

ABOUT THE AUTHOR