APM & SEE EXECUTIVE UPDATE VOL. 18, NO. 4
To technologists, technical debt, whether incurred intentionally or not, is an inexhaustible source of frustration and failure. But to technology-dependent organizations, unrestrained technical debt is a threat to survival. Many now recognize that if present trends continue, the more important controversy may not be whether we will suffer a technical debt-driven collapse, but when.
For technology-dependent products, companies, institutions, and even societies, sustainability depends on learning how to manage technical debt. Like most transformations, incorporating new practices into our organizations will likely be an iterative process. We already recognize the problem, and researchers are making progress, albeit mostly on technical issues. Nevertheless, some have begun to examine the nature of the required organizational transformations.
Little is known now about what those transformations will entail. So far, the literature of technical debt has emphasized code and other technical artifacts. However, because the root causes of technical debt may not be technical at all, the solution might be far more fundamental than changing how we build software. The required changes may affect how we sponsor projects, or how we account for expenses, or even what we mean when we declare that a piece of work is “done.”
To paraphrase Stephen Covey’s description of a jungle safari, we’re cutting our way through the undergrowth, staying focused on sharpening our machetes, conducting muscle development programs, and improving work schedules. Yet it might be time for someone to climb a tall tree and verify that we’re cutting our way through the right jungle.
This Executive Update proposes a policy-centered approach to the problem. It begins with a principle that can serve as a guide for constructing technical debt management policy, and then shows how to apply that principle to develop nine recommendations that enable organizations to manage technical debt effectively.
The Principle of Accountability
Organizational policy is of little use if gaining compliance requires the continuous focused attention of senior management. This is particularly true when the policy involves behavior related to technology and technology-rich decisions, as is the case with managing technical debt. The problem is too complex — and the decisions too numerous — for enforcement-based mechanisms to work. Effective technical debt management policy must provide a framework that supports new behaviors leading to rationalizing technical debt formation.
Nearly anyone who makes or contributes to decisions might occasionally bear some responsibility for incurring new technical debt. The organizing principle that leads to effective control of technical debt must be simple to state and easily understood because we must communicate it to nearly everyone in the enterprise. Here is a sample statement of that principle:
Those whose decisions or actions cause the enterprise to incur technical debt are accountable for securing the resources needed to retire that debt, and for supplying compensating resources to those within the enterprise, or among its customers, who suffer depressed operational effectiveness during the time that technical debt is outstanding.
I call this the “principle of accountability.” It’s a corollary of what Gerald Weinberg calls “Ford's Fundamental Feedback Formula,” which captures the idea that people make better choices when they must live with the consequences of those choices.
An alternative explanation of the effectiveness of the principle of accountability in managing technical debt rests on Robert Fritz’s principle of the path of least resistance. Fritz explains that people “go through life taking the path of least resistance.” As he puts it, “structure determines behavior.” Installing policy structure based on the principle of accountability would place squarely on the path of least resistance those choices that lead to effective technical debt management.
The two scenarios that follow illustrate the function of the principle of accountability. Each scenario has a description, from one person’s perspective, of a situation that leads to new technical debt formation. Each also includes commentary that describes how accountability and resources are allocated under a technical debt management framework (TDMF).
Do you know the true value of your software? Are you sure?
In a Technical Debt Assessment and Valuation, Cutter's Senior Consultants examine the quality of the software under examination through technical and business lenses. Whether that code is your own, has been developed by an acquisition candidate, or by a company you're investing (more) in, Cutter will help you answer the vital question "Is your software an asset or a liability?" Learn more.
For most organizations, these scenarios — for the time being — are far out of reach. Transforming the organization to support these scenarios would be a dramatic change. But that’s the nature of glimpses of possible futures: they’re unreal until we make them real.
Scenario 1: Development-Induced Discovery
As a software engineer, I want my code to communicate to the reader my understanding of what the code is supposed to do. In this project, during development, we learned much about the customer’s need, but too late to capture that learning in code in the way we now recognize was necessary. We want to return to the code after delivery and make it right.
Commentary
At the end of a project, if we recognize that our deliverable requires or could benefit from additional work that would facilitate maintenance or enhancement, we’ve incurred technical debt. This can be true even if the deliverable works correctly. We can then negotiate with the project sponsor to have him or her post an internal “bond” to secure the debt. That bond is a commitment to fund the debt retirement effort, using some of the sponsor’s future resources. Retiring the debt is in the sponsor’s interest because it improves productivity and velocity for future maintenance or enhancement efforts. Meanwhile, people who must interact with the deliverable in any capacity might suffer depressed productivity until we retire the debt. They can negotiate with the sponsor for resources to cover that marginal increased cost. They can also negotiate with the sponsor for coverage of the cost of revising their assets if needed when the new version of the deliverable is available.
If these negotiations reach an impasse, the parties can turn to the technical debt arbitration panel for a binding resolution. If that is necessary, the parties allocate resources to cover the work of the panel.
Scenario 2: Revenue Priority
As VP of strategic marketing, I’ve determined that we must establish, by the end of our second fiscal quarter, a strong presence in the cybersecurity market for the Internet of Things. The project sponsor and the development team have demonstrated a credible first offering. They have some misgivings about internals, and there are a few external blemishes, but we need to press ahead. The sponsor has agreed to trigger the release process, and I have agreed to press for a little funding for some post-release revisions.
Commentary
This scenario illustrates the tension between the need for revenue or market presence and the need for sound engineering. All parties understand that the post-release revisions will emphasize enhancing future development productivity and increasing future velocity. Still, strategic marketing might be expected to be reticent about some items on the list that will be developed by product engineering, test engineering, and release engineering.
If these negotiations reach an impasse, the parties can turn to the technical debt arbitration panel for a binding resolution. If that is necessary, the parties allocate resources to cover the work of the panel.
The Elements of the Transformed Organization
This section (from my forthcoming book Technical Debt for Policy Makers) contains a sampling of cultural and policy elements that facilitate the management of technical debt formation. They were developed from an understanding of the mechanisms that lead to technical debt formation, and which prevent technical debt retirement.
Legacy technical debt presents a different problem — one that can be addressed by more conventional means, as a series of projects. But it’s wise to defer, if possible, such efforts until formation of new technical debt is under control because uncontrolled formation of new debt could erase any gains made by retiring legacy debt.
The elements below are in skeleton form. The descriptions emphasize attributes that affect technical debt, but they don’t include many of the essential components of workable policy, such as definitions of standards, audiences, or responsible roles.
Excluded from this abbreviated list are amendment, appeal, and waiver processes, which are similar to corresponding processes for other policy structures, except of course that the adjudicators would necessarily require some expertise in issues relating to technical debt.
The discussion below also omits consideration of the path required to transform a given organization from its current state to one that includes the policies and cultural components suggested. It is nevertheless of some value because consideration of any organizational transformation must begin with the objective. In some ways, such insight is a form of Covey’s habit number two: “Begin with the end in mind.” Briefly, you’re a lot more likely to get where you want to go if you know where you are and where you want to go.
1. Adopt a Shared Concept Vocabulary
Because we cannot manage what we cannot discuss, managing technical debt responsibly requires a shared concept vocabulary. Some necessary terminology, such as the definition of technical debt, is obvious. Examples of concepts more easily overlooked are among the following:
- Done. Clearly define what done means. In project work, ambiguity about the meaning of done can lead to technical debt. As explained below, effectively managing technical debt requires steps that go beyond what most people now do. Those steps must be included in the definition of done.
- Deliberately incurred technical debt. Deliberately incurred technical debt is debt that we incur with intention. In code, knowingly using an obsolete method would be an example of deliberately incurred technical debt. Technical debt incurred deliberately for jointly accepted good reasons is strategic technical debt. Debt incurred deliberately by any means involving subterfuge or incompetence is reckless or possibly unethical.
Martin Fowler has created a 2x2 matrix classification of technical debt. The two axes can be characterized as the degree of wisdom (Fowler calls this “reckless/prudent”) and the degree of intentionality (Fowler calls this “deliberate/inadvertent”). Using Fowler’s Technical Debt Quadrant, we can rate an instance of new technical debt and use that rating when we set interest charges.
- Inadvertently incurred technical debt. Technical debt not deliberately incurred is inadvertently incurred. Following Fowler, inadvertently incurred technical debt can be either reckless or prudent. For example, some Excel 2010 macros became inoperative in Excel 2013. They constituted technical debt. Whether the debt is reckless or prudent depends on the situation.
- Development-induced discovery. Some technical debt arises because, in hindsight, technologists recognize better approaches after the work is completed. The methods they used aren’t wrong, but the work might need revisions for sound engineering reasons. Cutter Fellow Ward Cunningham’s original paper coining the term technical debt cites an example of development-induced discovery.
- Field-revealed discovery. Sometimes the users of an asset discover issues only after they start using it. These items aren’t defects because the asset does what the users expected. But experience reveals opportunities for improvement, and those opportunities can constitute technical debt.
- Total anticipated interest. An important factor in management decisions regarding newly incurred technical debt is its effect on productivity and velocity, commonly regarded as metaphorical interest charges for the debt. While the components of interest related to productivity reduction are beginning to be understood, our understanding of other possibly more important consequences is less advanced. These consequences include loss of market share due to delayed delivery, opportunity costs due to saturation of internal resources, increased help desk load, and more. Decisions regarding technical debt retirement should account for these ongoing “interest-related” effects.
- Debt origination and maintenance costs. Associated with home mortgages are various origination fees: appraisals, credit reports, document preparation, and so on. Technical debt also has origination fees, though we rarely recognize them. For example, careful “borrowers” document new technical debt to facilitate its eventual retirement. Some organizations maintain technical debt backlogs, which also generate costs. Technical debt origination and tracking is not free.
2. Accept That Technical Debt Is a Fact of Technological Life
Although we can avoid some technical debt, we cannot avoid it all. Some technical debt arises because of advances external to the enterprise, beyond its control. Development-induced or field-revealed discoveries are especially difficult to avoid.
The urge to retire all technical debt is likely due to using the word debt, and to deep societal biases against financial debt. Technical debt is not a form of financial debt. Carrying technical debt might not be evidence of a character flaw. In many instances, it’s an inevitable result of using technology.
3. Track the Cost of Carrying Technical Debt
The cost of retiring a particular class of technical debt is significant only when planning or setting priorities for debt retirement. At all other times, knowing that cost has little management value. What does matter is the cost of carrying that technical debt — the metaphorical interest charges. They can fluctuate wildly. Acquire and maintain expertise for estimating and tracking the cost of incurring and carrying each class of technical debt.
4. Assign Accountability for Classes of Technical Debt
Some technical debt results from actions within the enterprise; some does not. To control the formation of new technical debt from internal causes, hold people accountable for the debt their actions generate. Ratings of new technical debt using Fowler’s Technical Debt Quadrant can be the basis for assessing and distributing internal financial accountability for debt retirement and debt carrying charges. For example, a reckless/inadvertent incident could carry a rating that would lead to imposing higher assessed charges for the originators than would a prudent/deliberate incident.
5. Communicate the What, Not the How
Effective technical debt management policy communicates goals, and a framework within which they will be achieved, without prescribing explicitly how to achieve them. Technical debt management policies that specify criteria for making decisions are unlikely to work because of the complexity of modern technologies and the rapid pace of change. Moreover, specifying procedures for avoiding technical debt would be, essentially, a waterfall approach to the problem, requiring comprehensive knowledge of the technical debt situation space — probably an unattainable goal. An approach that empowers practitioners requires far less centralized situational awareness.
Below is an example of a policy that has little chance of contributing to effective technical debt management:
Decisions that require adaptation of technological assets shall be communicated in a timely fashion, so as to give those who must allocate resources, perform maintenance, or create enhancements a reasonable time to complete their work.
This example assumes that, given enough time, the work can be completed without incurring new technical debt. Unfortunately, that assumption is rarely valid for complex systems. The policy is simple and easily communicated, but it provides little guidance with respect to the technical debt problem.
A policy more likely to achieve the objective might be:
Teams making decisions that require adaptation of technological assets shall include in their deliberations representatives of the teams that will be performing the necessary technical work, including development, maintenance, and support. They shall negotiate a mutually agreeable budget and schedule for that work, including a plan for retiring any technical debt incurred.
This example policy relies on the autonomy of the practitioners to avoid incurring technical debt, or, failing that, to plan responsibly for retiring the debt they do incur. By including all roles in the decision process, the policy preserves balance between activities that might generate technical debt (development) and activities that bear the burden of that debt (maintenance and support). Note that, at present, planning for debt retirement is not usually regarded as a responsibility of the teams executing a given development project. Such planning is possible only if other policies support such a culture change.
6. Require That Deliberately Incurred Technical Debt Be Secured
In the financial realm, secured debt is debt whose repayment is guaranteed by specifically pledged assets. By analogy, secured technical debt is technical debt for which resources have been allocated to guarantee retirement.
Deliberately incurred technical debt, whether incurred strategically or recklessly, must be secured. If anyone involved in a development or maintenance effort feels that technical debt has been incurred, a dispassionate third party, unaligned with any function involved in the effort, reviews project deliverables for the presence of technical debt. Allocating future resources might require securing commitments of resources for fiscal periods beyond the current one. For many organizations, such forward commitments might require modifying the management accounting system.
7. Establish a Technical Debt Arbitration Panel
A TDMF depends on people negotiating with each other about allocating resources to secure or retire technical debt. Occasionally, when parties are unable to reach consensus, the technical debt arbitration panel works with them to find a resolution. Members of the panel have mediation skills and expertise in technical debt management for the specific technologies in question. Because the panel’s operating budget is in part covered by charges to the parties it assists, imposing these charges makes the parties accountable for failing to reach consensus.
8. Document Long-Term Strategic Technical Debt
Because of staff volatility, failure to document long-term technical debt leads to loss of understanding of what it is, why it is debt, why it was incurred, why it is long term, where it is, and how it should be retired. Documentation captures this information.
9. Adopt Technical Debt Ratings
Debt that has high interest costs must be rated so high that it is retired immediately. For example, debt in any assets related to cybersecurity would probably have associated with it a high interest charge because the cost of failures can be high enough to threaten the entire enterprise. Eric Allman recommends keeping “a journal of ‘debts that must be repaid before release.’ ” This very prudent recommendation might be too lax. A more conservative approach might include a feature that literally blocks release unless a condition is satisfied within the asset.
Final Thoughts
Managing technical debt is unlikely to be successful if we rely on enforcement approaches alone. Similarly, success will likely remain elusive if we rely on technical approaches alone, or policy approaches alone. Success will likely be based on finding a workable combination of all three: enforcement, technology, and policy.
The prevalence of technical debt in the technological assets of many organizations suggests that the current combination of these three approaches is not yet ideal. This is unsurprising because, of the three, only technology has received much attention. This condition is itself a form of debt — in this case, organizational debt. Like retiring technical debt, retiring organizational debt is an iterative process that involves gradually increasing our understanding. This Update has sketched the goal, in the form of some attributes of policies that help control technical debt. The form of that goal for any given organization will certainly differ. To discover that unique form, one need only make a start.
