"Almost everyone has access to far more data in a much more timely fashion than they used to (well, almost everyone), but is it better, more actionable data?"

Opening Statement

It has been just over 50 years since preeminent IBM computer scientist Hans Peter Luhn coined the term "business intelligence." And ever since then, BI has been viewed as getting information to the people who need it in a timely fashion and in a form that is easily consumed and acted upon (the right data to the right people at the right time). From those seemingly prehistoric days of data processing, when BI consisted primarily of monthly reports on green bar paper, to today's splashy interactive graphics on wireless mobile devices, both the data that is available and the means with which to deliver it to the right people have changed dramatically.

But have these changes made a measurable impact in organizations, or have the results fallen short of the promises? Almost everyone has access to far more data in a much more timely fashion than they used to (well, almost everyone), but is it better, more actionable data? In a real sense, BI tools, and the data hubs, data warehouses, and operational data stores that feed them, expand our view of data in the same way that a more sensitive telescope can see deeper into the universe. We now have the ability to see things in detail that we could never even see before; we can see deeper into the data on which our enterprises function. The real question with BI today is, are we seeing the right things at the time we need to see them?

Over the last few years, I've worked with a number of clients using a variety of BI tools. In many of those organizations, the "enterprise-level" BI tools seem to be limited to a small number of users, predominantly in the area of financial services. I worked with one organization a few years ago that had spent several million dollars on a BI solution only to discover that, once implemented, it had only 20 users in the entire company, and, of those, only a dozen or so were regular users. BI tools seem to be most often used to monitor costs and other predictable key performance indicators (KPIs), such as inventory levels, repair times, and other largely cost-related metrics. Occasionally, they are used to monitor value chains, such as supply and fulfillment chains. Once again, their use seems to be mostly limited to predictable metrics around spotting bottlenecks and trouble spots in the processes that are being monitored. On the flip side of the cost equation, BI tools are occasionally used to actively monitor revenue and growth forecasts and alert people (and, rarely, other systems) when problems seem likely to occur. Again, in most cases, the actual number of users in the organization is very small.

For many years, BI vendors have sold the promise of the "executive dashboard" -- a one-stop, all-encompassing view of the enterprise in which senior executives and decision makers are presented a graphic set of dials and charts they can use to instantly see and monitor the health of the organization. In this vision, any time an executive sees a red light appear, he or she can simply click on the widget to drill as deeply as necessary into operational numbers. The problem is, these dashboards seem to be few and far between in the real world. In fact, organizations have built executive dashboards in the past only to find that the executives don't use them. And that seems understandable: although portions of the enterprise can be distilled down to meaningful dashboards with a few KPIs, a large multinational private enterprise (or large public sector entity) is far too vast and complex to be translated into a handful of gauges and dials colored "red," "yellow," or "green." If it could, just about anyone could run it by simply managing the dials.

Perhaps, as one of the articles in this issue asserts, the next generation of BI should be actually "driving" the KPIs. As Kas Kasravi points out, we already do let some systems autonomously manage low-level systems, such as cruise control and autopilot systems. Some of today's systems do the same thing in financial equity trading: the so-called high-speed trading systems spot emerging market trends and make automatic buy/sell decisions in milliseconds. (In fact, these systems are colocated as physically close to the markets as they can be to eliminate latency due to the speed of light.) As you will read in Kasravi's article, there are going to be ongoing trust issues with letting BI systems "drive," as evidenced by the so-called flash crash that took place in May 2010. As you probably know, it seems that multiple high-speed trading networks all decided that a particular afternoon was an excellent time to sell, which of course led to other systems deciding it was time to sell, and so on. After the Dow Jones average dropped and subsequently recovered nearly 1,000 points in a few minutes, many were left wondering whether high-speed trading BI algorithms should be allowed to "drive."

For many years, BI vendors have also been promising meaningful analysis and integration of unstructured data. The storage, categorization, and retrieval of documents, e-mails, and other unstructured data have spawned an entire industry of content management, which many would consider to be a subset of the more generic BI. And in this exploding era of social collaboration, organizations are starting to use even more unstructured data from social media to enhance and support their BI capabilities. Ranging from mashups, to Web 2.0/3.0, to on-demand/cloud offerings, these tools may improve organizations' BI for better data analysis, forecasting, and collaboration. Yet while some initial results seem promising, the integration of unstructured data into BI systems remains elusive for most organizations.

Perhaps since the idea's birth in the late 1950s, we've come to expect too much of BI. I am reminded of a conversation Ken Orr and I had with T. Capers Jones back in the 1980s. His research at the time had studied the growth of data processing and projected out the personnel needs for the next couple of decades. The numbers told an interesting tale. They suggested that by the year 2000 (some 15 to 20 years in the future from then), every person in IT would have to be a computer programmer, or there would simply not be enough resources to satisfy the projected demand for new systems. He then pointed out that AT&T had done similar research at the turn of the 20th century, and back then it appeared that everyone on the planet was going to have to become a telephone switchboard operator in order to support the rapid growth in demand for telephone services up to that time. The problem, of course, is that technology continues to change. In a very real sense, everyone today is a telephone switchboard operator when we directly dial someone we wish to speak with, just as everyone in IT is a programmer when we create and manipulate spreadsheets, produce presentations, build reports, and prepare complex documents.

It seems that compared to the late 1950s, much of the original vision of BI has been realized in most organizations, and we have been so immersed in the changes that have taken place that the everyday satisfaction of BI needs often doesn't feel like true BI anymore. An analogy to warfare has been often used: when you are on the ground fighting a battle for the next hill, you don't see the hill beyond it until you've overtaken the hill immediately in front of you. Only then can you see the territory beyond and realize that although the battle for this hill is over, there's another, perhaps larger, hill looming in the distance. Our expectation that BI systems will be the be-all and end-all in providing everyone in the organization with the right data at the right time is really what's been happening in small measures all along. The only reason we realize that there are unfulfilled goals of BI is that we are now standing on a different hill, looking to another distant goal.

In this issue we have five articles, with views from several of the BI hills and a couple from some of the valleys. We begin with two of views of possible future aspects of BI. In the first article, Kas Kasravi uses principles from the TRIZ methodology (TRIZ is a Russian acronym for "Theory of Inventive Problem Solving"), to predict some of the characteristics we're likely to see -- or in many cases, not see -- in the next generation of BI tools and techniques. As you might expect, current best practices are only beginning to approach the levels he describes (i.e., the "driving" I alluded to above), and before they become mainstream, there are some interesting social challenges that must be overcome. Rather like many of the capability maturity models, his analysis not only provides a nice framework for analyzing the state of BI at your organization; it also may suggest directions in which it might be extended.

In our second article, Cutter Senior Consultant Bhuvan Unhelkar and coauthor Amit Tiwary make the argument for extending BI capabilities in a slightly more specific direction: what they term "collaborative intelligence" (CI). Whereas BI is sometimes narrowly viewed as a way for an individual enterprise to make more effective use of its information, their CI model envisions a broader view that leverages collaboration and information sharing across organizational boundaries. As proposed, CI would enhance BI capabilities for collective value and to ultimately reduce costs. Unhelkar and Tiwary describe the tools and techniques by which CI can be implemented, postulate a maturity model for the practice, and discuss some of the social and technological challenges that organizations may face.

The third article in our issue is an introspective "back to the future" view of business intelligence. Jan-Paul Fillié returns to Hans Peter Luhn's seminal paper on BI from 1958 to see how much of that original vision has been realized by BI implementations through the years. I was frankly surprised (and impressed) to find that a considerable portion of the architecture of current BI systems was correctly anticipated and identified back when the concept of data processing was still in its relative infancy. Perhaps equally interesting are the architectural concepts from that vision that are still missing or incomplete in today's implementations.

Next up is a change of pace. Ralph Menzano writes about some of the changes taking place in the area of BI for the transportation industry. Although the article examines some of the analytics and KPIs that transportation organizations might consider in today's marketplace, the observations about appropriate BI initiatives are relevant to many other industries as well.

To wrap things up, we have an article about business intelligence from a more personal perspective. Martin Bauer writes about the trials and tribulations of one man's search for meaningful BI in his organization. Although coming from the perspective of a small business owner, I found myself identifying with many of his frustrations, and I suspect you will as well.

Were he around today to read them, I think Hans Peter Luhn would be greatly impressed with the way these articles describe and extend his original vision of business intelligence. BI is indeed not only alive and well, but also being used in ways that the early pioneers in the field could barely have imagined. It will be interesting to see what the next 50 years hold in store. We hope you enjoy this issue of Cutter IT Journal.

ABOUT THE AUTHOR