CUTTER BUSINESS TECHNOLOGY JOURNAL  VOL. 31, NO. 7
 

Remember the “I see dead people” meme from the movie The Sixth Sense? As an architect practitioner, I see dependencies everywhere — sometimes they are empowering, but more often, they are implicit and constraining. Now, I don’t mean dependencies visible on the surface (i.e., visible in workflows, impact maps, package dependencies, and intricate UML diagrams). Rather, I am referring to dependencies found in the underlying (and subtle) living fabric of the business technology ecosystem (i.e., the implicit architecture, the network of interactions and delivery constraints, and the pulse of the runtime infrastructure); in short, the emergent dependencies.

Over the years, I have found dependency-oriented thinking a powerful tool and dependency awareness an easy common denominator among the different players in the architecture lifecycle. So why can’t we build a dependency-based model that is flexible, deep, and broad, and, at the same time, enable architects to answer their most common questions? Thus, I propose a dependency-based, lightweight, pragmatic approach to build architectural insight into the Agile delivery process and continuously reflect the changes inherent in the Agile process.

Despite the multitude of architectural frameworks and methods, experiencing a smoothly working, pragmatic synergy between delivery teams and the architecture discipline is rare. Root-cause analysis brings our focus to semantic gaps (i.e., gaps arising from the continuous erosion of contextual understanding in the development process). While common architecture practices address these via process and control, often such practices produce yet another layer of confusion in the organi­zation. Under the increasing pressure of accelerating marketplaces and rapidly evolving technologies, internal velocity and responsivity become significant differentiating factors. Indeed, responsive, flexible software ecosystems enable high-speed businesses.

This article describes one way of establishing a non-blocking architecture governance practice for Agile development teams. The approach consists of a few independent ideas that lead to organically integrating architecture into the process, the delivery pipeline, and team routines. This article also provides insights into common difficulties in Agile projects — the difficulties of capturing the right level of abstraction, of keeping a pragmatic balance between documentation and the rapidly changing deliverables, and, lastly, of integrating architectural practice into everyday teamwork. In Agile principles, communication and contextual understanding are key, but when it comes to architecture, being aware of tradeoffs and consequences has at least the same level of importance. The emergent approach proposed in this article is easy to introduce (even partially), easy to follow, and easy to adapt to varying team cultures.

A Glimpse into the Agile Architect’s Day

Given an established enterprise with its decades-old IT department, processes, and practices versus the accelerating marketplace — when missing out on modern IT practices and being too rigid to react to market trends, with even innovation on half-yearly cycles — then¹ we see the hiring of a talented Agile architect to bridge the gap and lead the recently established digital pillar of the company.²

Let’s explore the common challenges The Architect faces via the story of a day.

At the start of the day, The Architect’s mind is still full of the escalation emails that arrived last night. Most of them asked about viability and for accurate estimates of important future roadmap items. The rest ranged from a gentle reminder of the yearly roadmap planning cycle through blocker escalations to the postmortem details of a new technology that failed in production. The Architect is determined to prioritize the most pressing blockers first to enable as many teams as possible. A few quick decisions are made — hesitantly — under time pressure and not having enough information. They are guided mostly by gut feeling and instinct. A few responses to escalation emails get sent, asking for more business context in hopes of avoiding the invisible landmines of corporate politics.

Feeling more comfortable, The Architect joins in on early team discussions only to learn that there is no mutual understanding between the implementers and consumers of an API and that another team has no idea of the revenue impact of the long-awaited feature it committed to deliver. Without the right governance, this feels like herding cats, but an over-regulated process would slowly strangle the teams’ progress. So The Architect quickly advises the teams to talk, share knowledge, and remember to document it, trying to keep in mind that the architecture metamodel — the tool mandated by the corporate enterprise architecture group — needs to be updated, too. In The Architect’s rush to the governance board meeting, reports based on “accurate” figures are quickly put together and some draft slides are produced from memory, not having the real-time insight or tools to produce them on the fly.

At the architecture governance board meeting, there are fewer people than usual, and The Architect quickly scans those present to sense the changes in power around the table. The Architect knows that important decisions need to be made. But these decisions often reflect the typical uninformed conversation where everything has to be “just right” immediately and are often based on “the single objective truth” that people so blindly believe in. The Architect then quickly sinks into familiar thoughts: how the digital department of the company should design “good enough” flexible solutions instead of producing report after report; and how quality, context, and meaning can be so detached from the measures demanded. Becoming somewhat frustrated, The Architect focuses on the meeting again to quickly note a few important changes in direction, the roadblocks arising from corporate policy changes, and the awkward technology solution that a previously unheard of unit has just forced into production.

It’s now time for coffee with the team leads and fellow architects and to convey the most important relevant changes. This is also the appropriate time to gently shape the practice and understanding of the architecture. After a quick argument with the most knowledgeable old-timer — a strong analytical thinker who questions anything not accurate to the most minute detail and who has been reassigned from a monolith company to spread domain knowledge — it becomes apparent that a balance between design and engineering thinking is still far away. More hands-on architecture practice is needed, and a pragmatic evaluation of the next emerging technology stack might be a good candidate.

The Architect’s closing thought of the day is the need to catch up with the teams as early as possible the next day, as there is no time for activities such as capturing, rethinking, or redesigning the architecture or solutions. There is only enough time to quickly align the delivery teams in-flight to avoid wasted effort.³

Architects spend their days trying to synthesize the old and the new; the static and the dynamic; organic solutions and problems; and, ultimately, dealing with people, process, and architecture equally. Without the necessary contextual insight, usually from more than one context, striking the right balance is extremely difficult, leading to fragile tradeoffs.

It’s Not the Trees

“Architecture,” for the purposes of this article, is the interconnected structure of relevant people, roles, and visions within a certain context. Defined in this generic way, architecture always exists and is implicitly defined. Consequently, architecture expands through space and time (space being the context for the usual structural representations; time being the context for architecture dynamics). As such, our definition of architecture is far from static diagrams and docu­mentation; rather, it is the living fabric in the business technology ecosystem.

When approaching an architecture representation, a key point is its decomposition into elements, usually leading to a containment-based representation structure. We might then navigate the architecture along the “containing” relations,⁴ or using predefined viewpoints. “Navigating along the containing relations” is the ability and constraint that navigation is only possible along hierarchies in the usual drill-down order. By principle, this then constrains what can be seen and in what context.

My proposal is free navigation across layers, taxonomies, and granularity levels. Viewpoints by their very nature define a single context; thus, views might miss an indirect relationship completely. Viewing an organic system only via tree-based representations is akin to the “blind men and the elephant” parable. The multitude of different aspects without a holistic view can be misleading.

To fill the gap, the proposed model should represent transitive relationships and their projections to groups of elements. The model should be navigable in a spherical manner, not limiting the insight to specific taxonomies. Any subset of the architecture should be decoupled from the way it is projected onto views. Forcing the emerging relationships into predefined taxonomies should be avoided.⁵

To reach the ideal balance in governing architecture, the challenge is to harmonize the intentional and the emergent architectures. The former is driven top-down, along tree structures, while the latter relies on loosely structured, bottom-up information flows. Thus, to incorporate both, the core representation model needs to expand accordingly. Moreover, to represent architecturally relevant information, the model should center around structural properties and patterns tagged with additional information about relationships and elements. Cohesion and coupling, the most common structural properties describing component autonomy, need to be explicit. Information about specific elements is naturally reflected in the context defined by their relationships to their neighborhood. In summary, the simplest model is an unconstrained dependency network tagged with metadata on its elements and relationships — the Architecture Knowledge Graph.

Time Is as Important as Space

Similar to the semantics of user stories, which always express change, it is beneficial to describe architecture as a series of changes — structural changes through time. The smaller the gap between the architecture representation of the architecture and its realization, both in expressiveness and timing, the more relevant the architectural changes.

Focusing on Agile organizations, we can consider what might be the natural expression of architectural change through time. With Agile, changes are described through backlog items; therefore, it is a natural extension to define architectural deliverables in terms of epics, stories, and tasks.⁶ There are two specific concerns here: (1) the architecture items should use a specific language to support translation to the Architecture Knowledge Graph; and (2) architecture deliverables should precede their respective backlog counterparts.

The idea is to capture architecture epics, stories, and tasks one abstraction level earlier than backlog counterparts. Architecture epics are defined during roadmap planning. Architecture stories align with the iteration preplanning timeline (e.g., some iterations ahead), and architecture tasks are those activities performed on the smallest iteration scale. These descriptions feed into the Architecture Knowledge Graph, expressing either architecture definition or change reflected in the living fabric. While there are many architecture and project management tools based on the same idea, their base model is usually relational.⁷

Ideally, the architecture descriptions follow a simple unified language that can serve as a baseline to feed the architecture dependency model. Of all the architecture description languages, perhaps the simplest one would be a flavor of Gherkin.⁸ Custom statements can support the selection of a subgraph (given), the intended future context of a subgraph (when), and the expected architecture transformation (then) on the graph.⁹

DevOps to the Rescue!

While the higher abstraction-level definitions can now be fed into it, the Architecture Knowledge Graph is still missing the bottom-up, emergent information about dependencies. This information becomes context­ual in the network of the already-defined, higher-level context from the backlog and architecture design. Looking at the maturity of DevOps practices, plenty of data sources are available. We can easily imagine services, for example, running in a cloud platform surrounded by continuous monitoring, log analysis, and operational insight tools. A few examples include:

• Hypervisor and resource scheduler insights

• Cloud infrastructure metrics via the administrative APIs

• Networking insights available from common monitoring tools

• Application event logs and machine logs on individual instances

On top of the operational insight data, a continuous integration (CI) pipeline provides invaluable information about the static structure of the solutions as well as their behavior under test. This information includes:

• Existing components and their external/internal dependencies

• Existing architecture structure

• Environment configurations

• Static code analysis results

• External assets

We might find it challenging to feed this vast amount of information into the graph. Keeping the original purpose in mind, the continuous stream of low-level metrics is secondary to metric aggregates revealing potential dependencies. That said, nothing prevents the integration of an operational insight tool with the graph while the actual log aggregation and stream analytics workloads are managed separately.

With the intentional and emergent information in place, let’s see what questions we can answer with insights provided by the Architecture Knowledge Graph.

Actionable Insight into the Living Fabric

This approach invents no new concepts; it is a natural consequence of considering architecture within its broader context, from the running infrastructure to the roadmap vision. Compared to the broadly available architecture tools, let’s review a few differentiators in the Architecture Knowledge Graph:

• Its definition emerged from the jobs to be done, which architecture should fulfill.

• Its design principles follow those of viable, organically evolving systems.

• It doesn’t represent architecture as a set of static structures but rather as a continuously evolving system.

• It supports on-demand contextual queries (using a graph query language) without restricting predefined views or aggregates.

• It avoids using the concept of time and instead relies on ordering.

• It explicitly differentiates global- and local-scope computations; it does not try to use a single model for all.

In practice, architecture is better observed as it evolves, as opposed to freezing it and being left with a static structure. Dependencies are not limited to the explicit package or solution dependencies. Dependencies can be enablers and constraints at the same time. For instance, resource availability and networking are constraining dependencies but are also enablers of computation and communication in a running system.

Assuming the dependencies imposed by the IT strategy and the relevant subset of the existing architecture are captured, we can use the repository during roadmap creation to understand all possible directions and which ones are of high impact. Decisions can be easily communicated to the teams via the epics and stories and by interacting with the repository, pulling the relevant aspects. As the team progresses, unintentional dependencies emerge and the ones to be satisfied are checked. When coming to a decision point, the what-if scenarios can be checked based on the existing fabric. Finally, timely reports can be extracted on demand to demonstrate alignment and progress in the delivery of the architecture targets. People in different roles can easily and continuously extract their respective insights. With the ability to identify structural and temporal patterns, best practices can be fine-tuned.

Lightweight Architecture Repository

To summarize, in order to support the high-level scenarios, the core software solution needs to:

• Keep track of architecture definitions (i.e., epics, stories, tasks)

• Keep track of dependencies

• Make definitions and their dependencies searchable

• Make it possible to define cross-layer aggregate views

Thinking about the simplest possible solution, a graph database comes to mind. The global metadata and element search is better served by a specific, freestyle search component leading to the following components (see Figure 1):

• Dependency database — graph database (i.e., graph storage)

• Metadata database — search engine (i.e., searchable metadata storage)

• Dependency ingestion API — multiple APIs, layered by throughput and granularity (i.e., CLI API — verification, feed data, etc.)

• Definition API — DSL¹⁰ and import logic (i.e., CLI API — monitoring streaming data, CLI administrator, etc.)

• Visualization — integration with any diagramming tool already at hand¹¹ (i.e., Web API)

Some Implementation Traps

Global aggregates are notoriously difficult to compute on graphs; they should instead be precomputed in a reactive way. As the time dimension translates to ordering relationships in the repository, capturing staged architecture transitions requires extra attention in order to preserve the previous, current, and forthcoming versions of the changing elements. The DSL, used to describe the dependency graph changes, can be easily over-engineered while trying to explicitly represent all low-level detail. Querying graphs might lead to a longer learning curve.

Organizational Adoption

Teams can continuously work on the repository and use it to capture insights; at the same time, the solution is well suited to an internal open source approach. In this way, the architecture team can effectively shape the solution to the given technology stack and existing practices. Thus, the graph organically grows with every piece of discovery and analysis and with every architecture backlog item.

Emergent Maturity

Without being overly prescriptive about evolving dependency awareness and architecture maturity, there are clear markers of adoption. With increasing context­ual insight, teams are empowered to understand the architecture tradeoffs and clearly see the architecture constraints. Controlling the fabric of cross-contextual dependencies, architects can make good-enough, just-in-time decisions. Greater transparency results in higher efficiency in harmonizing the emergent and intentional architectures. The following four aspects help in understanding emergent maturity:

1. Introspection — describes how broadly depend­ency data points are captured and how easy it is to capture them

2. Emergent architecture — describes how much emergent information can be accommodated

3. Dependency awareness and usage — describes the extent of widespread dependency awareness and how much the repository information covers the overall software development lifecycle

4. Lightweight governance — describes how the practice of just-enough intervention, driven by contextual insight, is appreciated

We will now break down each aspect in more detail.

Introspection

• Ad hoc introspection is typically triggered by bugs/issues.

• We can capture CI and minimal operational insight, test results, package dependencies, and basic runtime metrics.

• We can perform static and runtime code analysis from local to production environments in CI, identify trends via comparisons to historical data, and use dependencies to drive testing on the infrastructure level (e.g., inaccessible service scenario).

• We can collect dependency data points in all environments through the CI and deployment processes; we can partially introduce control points.

• We can capture dependency markers through CI, deployments, and infrastructure; introduce dependency control points in all environments; use dependency management libraries during development; and correlate insights across the graph.

Emergent Architecture

• There is no concept of architecture, or it is considered isolated from the software delivery process.

• Architecture is a set of documents and processes manually updated and loosely aligned with the delivery milestones.

• Architecture repository, centralized view, manual or batch updates, and automated interactions are not fully supported.

• Architecture repository usage is partially automated; repository is navigable across different levels and supports on-demand contextual queries (i.e., free­style graph queries on the dependency-oriented core model).

• The Architecture Knowledge Graph, updates, and queries are automated and integrated into the lifecycle; on-demand contextual information is available; architecture descriptions and precomputed aggregates are supported.

Dependency Awareness and Usage

• There is no appreciation of the generic concept of dependencies.

• Only particular types of dependencies are observed and handled in isolation (e.g., solution and package dependencies and individual deployments with their resource demand).

• Classes of dependencies are identified and handled partially; the concept of delivery, logical, and infrastructure dependencies emerges.

• Dependencies are classified in alignment with the layers and structure of the architecture; this expands to the delivery and runtime infrastructure areas.

• Dependencies are proactively and intentionally managed and monitored; all architecture layers, the delivery process, and physical realization rely on the dependency graph.

Lightweight Governance

• There’s isolated or no architecture governance.

• Governance is based on team collaboration without the support of insights.

• A streamlined governance process is established utilizing a stand-alone central repository; team involvement is based on preliminary orientation meetings.

• A central repository is somewhat integrated, partial insight is available (e.g., batch refresh), and governance controls are manifested in the processes and team best practices.

• A complete feedback cycle through the layers of architecture,¹² delivery, and production sys­tems emerges. Architecture is controlled by the Architecture Knowledge Graph, its integration to the development lifecycle is automated, and dependency insights are continuously fed back.

In conclusion, in an Agile setup, we might just merely follow the flow to integrate delivery with architecture. But unlocking new information sources and continuously capturing relevant architecture details and just-in-time interactions with the Architecture Knowledge Graph lead to deeper insight and a contextual understanding. Contrary to common practice, introducing an architecture repository doesn’t have to be big bang; it can be incrementally put together with an enabling base model and some attention to dependencies. Following the improved insight, we can organically establish lightweight governance. Indeed, architecture is alive and continuously changing through space and time; static governance structures freeze it to fragility.

Architecture + Agile: The Yin & Yang of Organizational Agility 

This article is one of six in Cutter Business Technology Journal that explores how architecture can be the gateway to a future of continuous adaptation, innovation, and success for your organization. Order now with Code AA20 to Save 20%!

Outlook and Recommendations

While this article does not reference any of the following pointers specifically, they deeply relate to the dynamics and potential future of a balanced architecture governance practice:

• Barry O’Reilly’s concept of “architecting for anti­fragility” helps practitioners systematically address fragility in the process and solutions of architecture (see his article, coauthored with Gar Mac Críosta, earlier in this issue of CBTJ).

• David Snowden’s Cynefin framework gives significant insight into the observability and predictability of simple, complicated, complex, and chaotic systems.

• When considering the impact of observability on architecture, Cutter Consortium Senior Consultant Roger Evernden’s reflection about the learning cycle comes to mind.

• Category theory, a branch of mathematics, lays the foundations to grasp the expressive power and beauty of manipulating only nodes and relations.

• For the more demanding, analytical reader, Allen Woods’s portal, “The Performance Organisers,” provides a journey to structured coherent design, demonstrating linked information-based architecture modeling.

• Nicolas Figay’s pragmatic journey into using ArchiMate extended with graph capabilities (aka “enterprise cartography”) reveals many opportunities.

• Tom Graves is an infinite source of architecture wisdom, tools, and advice. More specifically, he gives advice on navigating the layers of architecture.

• Randy Shoup’s “Minimum Viable Architecture” concept is a real-life buoy when seeking the right balance.

• When thinking about sustainable systems, we cannot ignore Stafford Beer’s viable system model.

• Simon Brown’s C4 model (context, container, component, code) is a pragmatic method that supports emerging contextual architectures with a rapid learning curve.

• Finally, both Disciplined Agile Delivery from Cutter Consortium Senior Consultant Scott Ambler and the Scaled Agile Framework (SAFe) detail Agile architecture in the delivery context.