CUTTER BUSINESS TECHNOLOGY JOURNAL  VOL. 31, NO. 7
 

Nowadays, there is a huge popular demand for Agile as a means to enable change and accelerate value. Popularity, however, is something other than reality; for most companies, the introduction of Agile requires a significant mindset shift. This almost always meets resistance from several directions in the organization. In addition, Agile adoption is often accompanied by some element of inefficiency and chaos if left unguided.

In contrast, enterprise architecture (EA) suffers from a decreasing reputation in technology innovation. This reputation can be the result of certain poor practices: in most organizations we encounter, architecture mainly focuses on its function as a design authority or regulating body. In this sense, it is often seen as an inhibitor of change instead of an accelerator — the infamous “architecture police.”

This article describes in more detail these and other common pitfalls and bad practices and tries to identify the pros and cons of both Agile and EA to find ways that the strengths of one can help prevent the weak­nesses of the other. We begin by highlighting some bad practices or common mistakes when introducing and operating Agile and EA. These bad practices tend to appear and reappear and are difficult to root out; we therefore use the term “anti-pattern,” as introduced by Andrew Koening, to label the bad practices. Wikipedia defines an anti-pattern as “a common response to a recurring problem that is usually ineffective and risks being highly counterproductive.” Next, using real-life examples from client work, we show the advantages available to organizations that successfully combine Agile and EA.

Challenges in Enterprise Architecture

The past 30 years have seen the rise and maturation of enterprise architecture. Beginning in 1987 with John A. Zachman’s Information Systems Architecture model, the EA discipline has since seen many advocates, research, and frameworks worldwide. Its foremost promised value: improving the efficiency of IT assets and the return on IT investments by designing, improving, and managing the complex landscapes of information systems.

Nevertheless, over the last 10 years, several published studies and opinions indicate that the value of EA may be overrated, and that EA might not deliver on its promise or might even be considered “dead,” as Australian columnist Jon McLeod has stated. Further extensive research, such as from the Erasmus University Rotterdam, also leads to a somber conclusion that two-thirds of EA projects fail. Even though this is probably “at par” with all other IT-related endeavors, it is an aspect we need to improve.

As we take a closer look at the reasons why EA has fallen short of expectations, it is valuable to highlight two anti-patterns that we have encountered repeatedly with clients.

EA Anti-Pattern 1: Process over Value

The first EA anti-pattern is apparent when enterprise architects are more concerned with correct adherence to the architectural process, often defined by their own derivative of an EA framework (e.g., TOGAF). Of course, this process was designed to ensure the maximum effectiveness and value of architectural work, but the focus should always be on the value part of the equation. The tendency to simply follow the process, designed in the past and often inherited by generations of enterprise architects, is too strong.

In practice, the enterprise architects are then more concerned with the correct “paper trail” of architectural documents, design and solution architectures, and, the most dangerous TOGAF template of all, the “Architecture Requirements Specification.” This limits the effectiveness and responsiveness of the EA team and, most importantly, erodes the perceived business value of EA in general.

EA Anti-Pattern 2: The Disconnect of the Ivory Tower Architects

The second anti-pattern occurs often in large organi­zations where the EA function has been recognized as important but resides as a staff organization — typically part of the office of the CIO. Because of its “staff” label, the enterprise architects’ impact and mandate are limited; managers can avoid or ignore architects to a great extent. Especially when the informal culture is strong and decision making is, in effect, local and lies with project managers, the enterprise architects become increasingly detached from the day-to-day practice of projects, programs, investment boards, and even strategic discussions on the mid- and long-term future of IT. They do not contribute visibly to successes and failures and hence cannot be held accountable. When this endures, the enterprise architects will be practically disconnected from the (informal) IT powerhouse, will be unable to deliver solid and trusted advice, and will have no mandate over strategic IT discussions (see sidebar “The Ruins of the Ivory Tower”).

Challenges in Introducing Agile Practices

Agile is a mindset. The overall concept of Agile is about continuously getting better at whatever it is we are doing. The practices and ways of working related to an Agile mindset have been well proven as more effective and efficient than the traditional waterfall methods — across IT and other business functions alike. Not only do they help create more value, but they also cultivate more candid and authentic workplaces. Looking for benefits such as faster time to market, better-quality products, and more engaged employees, organizations of every size are now adopting and maturing an Agile mindset.

However, as companies adopt Agile as their standard for software development, they usually encounter resistance from several directions — from other parts of IT as well as from the business. This resistance is a result of aversion to change, with existing structures and leadership holding on to practices that worked in the past. Indeed, it is quite challenging to expand the first successful pilot projects to an enterprise-scale Agile capability. We often see organizations struggling with cultural change, insufficient business involvement, and other aspects of scaling. In design thinking terms, these challenges are called “hills.” The hills model of Agile transformation (see Figure 1) shows the most common challenges that organizations face when applying Agile-at-scale; note: the order in which these hills are encountered is different for each organization:

• Hill 1: Changing the organizational culture. Changing the organizational mindset is key for a successful Agile implementation on an enterprise scale.

• Hill 2: Getting the business involved. The role of product owners, but also the support of the overall business, is the driving force behind Agile success.

• Hill 3: Coping with different speeds of change. Not all parts of the business and not all teams providing support for technology can work and accelerate at the same pace.

• Hill 4: Extending Agile to the full lifecycle with activities and automation. Operations can only embrace Agile if nonfunctional requirements are met. This can be done through end-to-end enablement and automation.

• Hill 5: Scaling Agile. Collaborating on an enterprise scale requires finding alignment between different teams to cope with dependencies, to share best practices, and to distribute the work effectively.

• Hill 6: Distributing Agile. Distribution allows for access to talent and resources across the globe, potential cost reduction, and opportunities for improved innovation.

To overcome these hills, some organizations use ways that worked for them in the past, but in an Agile context this results in counterproductive outcomes. As we saw earlier with EA, these anti-patterns are hard to root out and tend to reappear. Below, we highlight two of the most common Agile anti-patterns.

Agile Anti-Pattern 1: Self-Driven Teams Running Wild

For any organization starting with Agile, it makes sense to begin with one or more pilot projects to demonstrate the added value of working in this new way. Practices like retrospectives capture lessons learned and ways to improve. From this first initiative, the organization can then start expanding. Several organizations have found that dispersed initiatives do not necessarily bring added value to the overall business. If each department has its own Agile teams defining their own way of working and making technology decisions without consulting each other and without a common vision, then Agile can soon become a very expensive exercise with only local benefits for the business. After some time, this can result in unwanted internal competition, multiple standards, different ways of working, and an increasingly complex technology landscape.

Agile Anti-Pattern 2: Lack of Enterprise Alignment

This second Agile anti-pattern usually happens in larger organizations where architectural considerations are entirely left to the development teams themselves. Without proper alignment through technical and business architecture, making the best technical and architectural decisions at the product level can result in unintended negative consequences for the larger organizational ecosystem. This becomes especially important in enterprises with large application port­folios, and even more important when a mix of legacy and modern technology needs to be integrated. While the design and execution of each product on its own is very important, ignoring the context of the larger ecosystem where these products will be used is highly risky. Similarly, business processes will be difficult to align and overall cost will increase.

This anti-pattern is a result of pushing decision making down to small self-organizing teams without properly enabling them with the knowledge and support they need to get the job done and a clear business direction to guide them. Architectural spikes and refactoring are not enough because the team does not have sufficient understanding of the dependencies within their organi­zations. The consequences range from resources spent on overlapping and redundant functionality to significant amounts of rework needed to course-correct when issues surface.

Learning from Each Other: Bridging the Gap

The anti-patterns described above show a clear need for better practical approaches to both EA and Agile. Based on our client experience, a significant opportunity to improve the business value of both lies in combining their practices. Several frameworks could help in this sense, and with the right vision in mind, companies can increase their ROI for both EA and Agile. For example, the use of frameworks like the Scaled Agile Framework (SAFe), that are built around the idea of striking the right balance between Agile and EA, is growing significantly — although with various degrees of success, to be frank. The balance of “emerging design” and “intentional architecture” is key here, as well as the way the IT organization is structured in terms of roles, responsibilities, and level of cooperation.

In practice, we have seen several ways in which organizations combine EA with Agile thinking and methods to break through the anti-patterns and improve results. The next section highlights four useful examples.

SAFe: Providing the Long-Term View to Agile Teams

The first example is from a Netherlands central gov­ernment client and comes down to the adoption of Agile architectural approaches described in SAFe 4.5. This organization has adopted SAFe as the standard process and methodology framework for its journey toward more business agility and more efficient use of IT assets. Up until 2017, the organization’s IT had increasingly become a liability due to outages, aging software and subsequent support issues, and rising costs. The enterprise architects were dealing with both anti-patterns discussed earlier: they were part of the CIO office, with limited mandate, and were mostly concerned with compliance — both in terms of archi­tecture as well as of the business as a whole — around issues such as security and public safety.

A huge modernization program was launched in 2017 with the general idea of using a greenfield approach to modernize the corporate IT landscape. New appli­cations were going to be deployed, cloud-native, onto a new hybrid cloud: the innovation domain. The client’s own adoption of SAFe 4.5 was to govern the development and management of the growing port­folio of these applications. Existing applications were analyzed and, if deemed appropriate, migrated to this new hybrid cloud in a separate environment that supported multiple operating systems, middleware, and databases: the migration domain. The remaining applications that needed some sort of lifecycle extension due to business requirements were managed using traditional IT methods in the legacy domain. The rest (around 15%-20% of applications) would be terminated.

Over a period of five years, this ecosystem was then going to be simplified and, in effect, the legacy and migration domains would be phased out. Table 1 illustrates the domains over time, with their appropriate solution development and governance framework.

The role for architects in these domains (and over time) varies. In the innovation domain, the architects act in three SAFe abstraction layers: on the large solution, program, and team levels. They actively participate in the role of solution architects and solution engi­neers. They are responsible for the SAFe practices of writing enabler epics and participating in value stream coordination sessions to establish backlogs and architectural runways. They also oversee the non­functional requirements, the reuse of solution elements, and the establishment of an architectural repository, where solution elements and patterns are managed. Thus, the architects focus on value instead of process.

However, the top level of SAFe, the portfolio level, is not implemented. At this strategic level, the enterprise architects still must deal with the long-term IT strategy and their current political reality of negotiation, delays, and alliance formation that typifies government policy and decision making. They will keep relying on their current EA approach, largely based on TOGAF. There is a reason for this choice: it is proving quite useful, since during transition (2017–2022), there is a bimodal character of the IT landscape to be managed and the choice has been made to be pragmatic and use the existing way of working to allow the top level of the organization to get used to the cultural aspects of adopting Agile.

Growing their impact in this IT transition period, as illustrated here, will help the enterprise architects add value by identifying cross-team solutions, promoting the reuse of those solutions and patterns (standard­ization), building the architectural runways, and advising on prioritizing the backlogs on various levels. Moreover, the Agile way of working has facilitated more direct involvement of the enterprise architects with the teams. The architects are more engaged in day-to-day discussions and, because of the increased heartbeat of the IT organization, they have more touchpoints where they can advise and guide IT decisions. Their former stronghold was in effect destroyed for the innovation domain: there was less need for all the blueprints, project start architectures, and other documents and designs up front that they were concerned with before. This meant a shift in thinking and a shift in pace that has proved beneficial so far (the program is still in its turbulent midterm phase).

Architecture: Tool to Prevent and Resolve Technical Debt

Another example in combining Agile and EA comes from a company in the consumer sector. This organization had a group of enterprise architects that published guidelines, contributed to the IT strategy, and participated in board reviews. When the company started its first Agile/DevOps program for the rollout of a modern e-commerce platform, the early architectural decisions were left entirely with the technical leads in the devel­opment teams. This worked well for the first year, but as the program grew in scope and complexity, the company decided to assign an architect directly to the teams — one architect covering multiple teams — and to create an architecture competence center specifically focused on this program. Across the board, there was more attention to architecture and a standardized, reusable solution.

From a cultural perspective, this proved challenging because the architect’s role was unclear. The teams perceived it as a control function, since they now had to justify in detail their decisions and participate in biweekly architecture boards. While the cultural impact still needed to be improved upon, the architect contributed to more communication within the teams, more informed decisions, and a longer-term perspective, asking questions from different angles and bringing into the discussion aspects of which the teams were not previously aware. By working more closely with the development teams, the architect also benefited from the opportunity to observe and get direct feedback on how the architectural guidance and design impacted the execution.

Moreover, with a wider view of the enterprise landscape and direction, the architect was able to identify an architecture backlog and collaborate with the product owner to add and prioritize the relevant items from this architecture backlog into the product backlog. A positive outcome has been better management of technical debt. For example, the e-commerce platform originally had been set up in a data center. While the development team had recognized the need to move to the cloud and identified this as technical debt, the product owner did not give it significant consideration. After the architect joined the team, his input helped the product owner prioritize this technical debt item and the platform was moved to the cloud.

Streamlining Agile-at-Scale

One of the more successful adopters of Agile, Spotify, has created an organization where teams are allowed to define their own backlogs, select their own priorities, and, in some cases, develop overlapping or concurrent functionalities. A Benelux bank used the Spotify example as best practice in 2015. Using the Spotify model, the bank adopted Scrum as a standard across the organization and became an example for several other non-IT companies on how to implement Agile.

The starting point was the department that developed client-facing applications. The department’s personal banking app, developed by the first Agile team, was one of the first and is still one of the most successful applications of its kind. Other departments soon introduced Agile as well, and Agile became the de facto standard across the organization.

There was a downside, though. The adoption of the Spotify model left technology decisions, automation tool choices, use of frameworks, and even sprint durations to each individual team to decide. This resulted in more than two years of effort to rationalize the software catalog by tens of millions of euros and to drive programs and internal promotion campaigns to standardize on a common way of working and an enterprise framework for Agile.

Using this organization as an example, a competing European bank decided to do its Agile implementation slightly different. To roll out Agile across the business units, the bank defined an Agile program supported by architects. Within this program, the bank brought together expertise from different IT partners. Enterprise architecture developed an Agile vision for the organi­zation. The bank involved a consultancy agency to provide Agile coaches who trained internal Agile coaches to take over in the future.

One by one, entire development teams were trained in a common and always improving way of working, based on industry standards. Each team was supported by a solution architect and could reuse all building blocks from a growing architecture repository. The bank regularly captured and analyzed retrospective results and maturity measurements from a growing set of Agile teams and incorporated these in standards and supporting materials. Similarly, it selected and piloted development and testing tools, and distributed those to the Agile teams as well. All these measures greatly increased the adoption rate of Agile.

Architecture as a Servant Leadership Function

Our last example comes from our experience at a US technology company. This organization had a very strong EA function that acted as a checker and gatekeeper. When the company started adopting Agile ways of working, this EA layer got completely dis­banded, and architectural decisions were pushed down to the small cross-functional teams to facilitate bottom-up thinking. This proved challenging because it was impossible for any team of eight to 10 people to maintain a comprehensive view of this complex organization. Teams started overlapping in certain areas of responsibility and did not fully understand where they fit in the wider ecosystem. The risks of building functionality that already existed or impacting strategic decisions that the teams were not aware of was too high.

Consequently, this company started rebuilding an architectural function that was very different than the one they previously had. The architecture layer was now primarily solution architects with an enterprise view. T-shaped skills were a must for this new EA group, allowing the architects to be nimble and work closely with business executives, product owners, and development teams alike. Instead of the old architecture review boards, these architects with broad enterprise knowledge and deep technical skills in different areas were now organized informally in a guild.

While decisions continued to be taken by the lowest level visibly impacted, the new architecture function became a trusted advisor, ensuring the decision makers had the knowledge to make informed choices. In this case, the architects did not build an architectural backlog as in our consumer sector example. Instead, they worked very closely with the product owners and the teams to help them prioritize, make decisions, and choose the right funnels to allocate work. Thus, the architects became servant leaders in this organization, playing a central role in creating value by driving integration, enabling the Agile teams, facilitating commitment and consensus, and removing blocks.

Conclusions and Call for Action

Organizations suffering from either or both architecture anti-patterns can benefit from Agile adoption, thereby introducing a faster pace and facilitating more direct communication between the enterprise architects and the Agile teams. The architects must be willing to dive in at the team level, but, in that process, they will become more relevant and valuable to the organization.

On the other hand, organizations “running wild” with Agile and suffering from too much decentralized technical decision making can benefit from architectural thinking, such as business architecture and business prioritization, standardized technical building blocks, nonfunctional cross-team solutions, and backlog prioritization and planning. With solution patterns, standards, and best practices, architects can guide the teams and build a longer-term perspective.

Of course, knowing all this, the goal is to be aware of the value and possibilities that both disciplines have to offer and to implement them simultaneously to prevent pitfalls. It’s up to the CIO and the CIO staff to design an IT operating model that combines Agile and EA, considering the maturity level of both Agile and EA in the organization.

Finally, to answer the question in our title: it’s neither agile architecture nor architectural agility — it’s both!