CUTTER IT JOURNAL VOL. 29, NO. 3

THE SOMEWHAT-HIDDEN COST OF RAPID SOFTWARE INNOVATION

One of the defining characteristics of software innovation, as opposed to other types of innovation, is the speed and ease of production. Other than time and labor, the only resources needed are relatively inexpensive hardware and software. Inventing a new type of jet propulsion, industrial ceramic, Alzheimer's treatment, or oil extraction method requires far more expensive equipment and other materials — not to mention more time, skilled personnel, and, in many cases, extra effort to meet regulatory requirements — than building a new social media service.

Usually, this ease of innovation is cause for celebration. The entry requirements into the software market are lower than, say, the automobile market. The speed with which innovation can radically transform the way people work and play is so great that we have come to take it for granted and even judge companies like Apple on the basis of whether they can make every year an annus mirabilis of discontinuous product innovation. Even in the more prosaic world of software innovation in corporate IT, we urge companies to go much faster than they are in building and deploying new software systems and criticize them for not seizing the opportunity for profound "digital transformation."

There is a price tag for innovating this quickly and easily. One of the costly line items is technical debt, the increased drag on the ability to do software innovation that arises from a very specific source: failing to code with a proper level of care and diligence. Writing code is much like another procedural exercise -- writing laws and regulations. The less care one takes in the process of creating these instructions, the harder it will be to diagnose unintended problems and fix them, or even to build on the existing procedural foundations.¹ Ambiguous intent, excessive complexity, untraceable logical paths, untested scenarios, and other problems are the inevitable result of rushing into "production" both statutes and compiled code. In the case of software, these problems take the form of cyclomatic complexity, indecipherable naming conventions, inadequate unit testing, and other problems.

As Cutter Fellow Ward Cunningham, who devised the concept of technical debt, was quick to point out, all forms of technical debt are not necessarily bad.² For example, a software team might want to experiment with a new approach to cloud-based storage to help an application scale. In this case, dotting all the programming i's and crossing all the unit testing t's is not merited. If the experiment fails, the extra work spent on code review, proper naming conventions, and other steps needed to prevent future technical debt would be wasted effort. On the other hand, if the experiment succeeds, it is imperative that the team go back and clean up the code. Failing to do so will make it harder to either build on or fix that code.

Of course, we would not be talking about technical debt if teams took the time to refactor, as well as other steps needed to "do it right." Temporary measures become permanent, which imposes a burden on the team. The greater the technical debt, the harder it is to work with the code base, slowing down further software innovation.

What stops highly skilled, dedicated professionals from doing it right in the first place? Some technical debt is inadvertent, even unconscious: a developer fresh out of college, for example, might not understand how important clear naming conventions are and may need mentoring on how to avoid some common complexity traps when designing classes. However, a major portion of technical debt -- if not the vast majority -- seems to spring from a very well-understood, well-identified source: time pressures.

Here is where the ease of software innovation becomes a burden. Not only do time pressures compel developers to cut corners, but they become the chief obstacle to fixing the code. In the face of increasing demand from the business side of the organization (sales, marketing, and executives in software companies; internal customers and executives in IT organizations), which neither understands how software innovation works nor cares about the particulars of writing good code, developers are hard-pressed to justify investing the time in short-term refactoring, even if it has long-term benefits.

GRAPPLING WITH TECHNICAL DEBT

That very cursory description of the problem oversimplifies and overlooks many important details. For instance, we have not gone near the differences between technical debt and other kinds of debt, the different forms of intentional and unintentional technical debt, the specific coding sins that lead to technical debt, or the steps needed to both eliminate current debt and prevent future debt. On that last front, opinions vary widely, from "Run a static code analysis tool once in a while" to "Just don't write poor code." Fortunately, in this issue, we have many good articles that help flesh out these critical details.

Our first article, by Declan Whelan, provides a more complete overview of technical debt. Whelan is one of the cochairs of the technical debt working group for the Agile Alliance, to which I was also a contributor.³ While he argues that technical debt is more a symptom of larger organizational and team problems than a problem unto itself, he lays out a clear method for diagnosing and addressing technical debt, which he believes is a much larger burden on software innovation than most realize.

Our next contributor, Jean-Louis Letouzey, is the other cochair of this working group and a noted expert on technical debt. Letouzey provides an introduction to a very specific strategy for dealing with technical debt, the SQALE method. In his article, he explains how SQALE measures technical debt at both the system and portfolio levels. While elements of SQALE will be mysterious to some business users, both they and the software teams can look at the output of assessment tools that perform SQALE analysis to see the current amount of technical debt and understand (if only in the rough outlines, for business users) the burden it imposes.

John Heintz, a Cutter Senior Consultant who has helped many clients deal with technical debt problems, uses the cost of change as the starting point for his article. Heintz then lays out a strategy for using technical debt assessment as the basis for better technical and business decisions and offers examples of companies that have pursued this kind of strategy successfully. Echoing Cunningham's dictum that not all technical debt is necessarily bad, and certainly not all technical debt is worth removing, Heintz cites one example of a vendor that intentionally increased technical debt in the short term because that was the right business decision at the time.

Next, Richard Brenner warns against the risks of using the technical debt metaphor too loosely. In many cases, he argues, people have confused the metaphor with the reality, including situations where people look for objective measures of debt. Like other authors in this issue, Brenner believes that technical debt is a symptom of other maladies in the organization, and he provides a simple list of measures for preventing further technical debt accrual.

Ram Reddy then provides a picture of technical debt that goes beyond the boundaries of a single software system, which is often the implicit basis for describing and assessing technical debt. Not only does technical debt have an impact on other, related systems (e.g., on the applications that depend on a debt-ridden identity management system), but it also appears in the integration code that binds systems together. Technical debt exists throughout the services catalog, including the "shadow IT" systems that the IT organization struggles to bring into the official fold. One of the most damaging effects of technical debt from this IT services catalog perspective, Reddy argues, is the way in which debt locks into place systems that should be candidates for retirement.

In our final article, Mohan Babu K explains that not all technical debt results from developers typing on keyboards. Babu K expands the concept to include the drag on the portfolio and projects created by the failure to upgrade vendor-provided software. Over time, the cost of maintaining, integrating, and extending these systems increases if IT departments do not perform these upgrades. He cites the enterprise architect as someone well positioned to help organizations decide on the priority and timing of upgrades.

THE COST OF TECHNICAL DEBT CONTINUES TO INCREASE

As demonstrated by this very rich menu of perspectives on technical debt and the lively debate that discussions like these inspire, Cunningham struck a nerve with the technical debt concept. Whether you believe that the technical debt problem is larger or smaller than others might think, a symptom or a cause of organizational dysfunction, and/or an absolute or subjective measure of that dysfunction, it is an unintended cost of the ease of software innovation that organizations must take seriously. The sources of technical debt, such as the ever-increasing demand for new software, are not going away. The software landscape continues to grow in complexity, with new devices, frameworks, integrations, and other features. Therefore, we cannot afford to make the job of software innovation any harder than it already is.

ENDNOTES

¹ In a similar vein, musician Frank Zappa once said, "The United States is a nation of laws, badly written and randomly enforced."

² To hear Cunningham explain the nature of technical debt and his inspiration for creating the concept, see: http://c2.com/cgi/wiki?WardExplainsDebtMetaphor.

³ For the complete package of support materials that the working group has provided, see: www.agilealliance.org/resources/initiatives/technical-debt.