"Are organizations better off building BI solutions when they need them and as quickly as they need them ... without a standardized and integrated DW?"

Bulwark Against Chaos

Here we go again with another virtual solution! It didn't work for data warehousing -- what makes us think it will work for BI? We still have to address the data chaos in our organizations. BI without a DW won't get us there, but BI with a DW will.

Onward with Technology

Building and sustaining a DW is slow and costly. Businesses can't wait for it and shouldn't have to! SOA, EII, Web 2.0, and other new technologies enable users to build their own BI solutions directly against operational systems. Let them, if they see a business value.

Opening Statement

Almost every organization has data problems. These problems can be grouped into two categories. One type of problem is having redundant, inconsistent, and often plain wrong data. The other type is not having easy access to the data. From the inception of the data warehouse (DW), the objective of building a DW has been to solve both types of problems. First, data is compiled, standardized, and integrated into a DW as the "single version of the truth." Second, end users in the organization are given access to the DW to do their reporting and analytics, which has become known as business intelligence (BI). To this day, there is a large contingent of BI experts who see the DW as the engine (or "plumbing") behind BI.

However, as online analytical processing (OLAP) tools, report writers, BI analytics products, and even MS Excel have matured over the decades, so too have the end users and their requirements. Today's new requirements include near-real-time access to operational source data, as well as very short fulfillment time of end-user requests. The DW community is trying to address these new requirements by building operational data stores (ODSs) and changing daily DW batch loads to near-real-time trickle feeds (i.e., ongoing streaming of source data into the DW databases). Unfortunately, it is often the case that neither process can be built fast enough, not to mention maintained and expanded fast enough, to satisfy the end users. As a result, end users frequently help themselves. They buy their own BI tools, maintain their own servers, get access to operational source data, and build their own BI solutions -- all without a DW and without IT.

This raises the following questions: Are organizations better off building BI solutions when they need them and as quickly as they need them, without the overhead of a costly and complex DW? Are the risks an organization takes acceptable or unacceptable when end users add more redundant, inconsistent, or plain wrong data by building silo BI solutions without a standardized and integrated DW? Who in the organization, or what types of situations, should determine which BI/DW strategy to pursue? After all, there are good arguments on both sides.

THE CASE FOR BI WITHOUT DW

Nobody disputes that bringing order into chaos is desirable, but building a DW for the entire organization takes a lot of time. The more data there is to be standardized, the more departmental views there are to be resolved. And the smaller the DW team is, the longer it takes to build, maintain, and expand the DW. End users, especially at the operational and middle-management layers of the organization, cannot wait for the DW team to catch up with their BI requirements. In addition, DW teams themselves have been promoting the idea of user self-sufficiency for a long time, acknowledging that the DW teams cannot possibly get to all the user requirements in a timely fashion. Therefore, there should be no reason why end users should not take advantage of the modern BI tools and build their own BI solutions.

For example, today BI applications can easily be created by power users with MS Excel, a tool many end users are very familiar with and love. Excel's Multiple Source Simple Output (MSSO) coupled with Structured Data for Excel (SD4E) is capable of quickly and easily manipulating and analyzing live data directly from multiple, disparate operational data sources. MSSO's ReadyData Reporting Tool provides end users with a wide variety of personally customizable dashboards and templates. In addition, MSSO's DataGuard protects the original data from data corruption as well as unauthorized access. Excel and other specialized BI software offer easy point-and-click capabilities for extracting data directly from the operational source systems without the need for a DW, loading the data into personalized databases, and providing analytical capabilities. For many end users, it's a dream come true.

THE CASE FOR BI WITH DW

On the other hand, one could argue that BI without DW is simply an excuse to create silo point solutions with no coordinated data standardization and integration effort among the end users who want to build their individual BI applications. In addition, while end users frequently claim that they know their data best, know where to get it, and know how to manipulate and analyze it, these same users do not agree on what the data means, what business rules and policies the data is subject to, and how to use the data properly. Each business unit or department has its own view of the data, which collides with many other views from other departments. The purpose of a DW is to standardize the data in order to provide a consistent and trustworthy single version of the truth for all end users throughout the organization.

Moreover, why should data be treated differently from any other organizational asset? After all, organizations apply one consistent set of standards, policies, and rules for managing human resources, equipment, buildings, parts inventory, financial assets, and so on. No organization in the world would allow each department to develop its own financial chart of accounts or to use a different salary structure and benefits policy from the corporate one. So why allow each department to create its own set of financial reports from its own private databases?

IN THIS ISSUE

For this issue of Cutter IT Journal, we asked practitioners and consultants in the BI/DW space to weigh in on the question "BI: With or Without DW?" The results were very interesting. While all authors to some extent acknowledged the benefits of data warehousing, opinions were split about BI's necessity for, or reliance on, a DW. Almost all authors addressed the challenges of building, maintaining, and expanding a DW, regardless of whether the DW is a central enterprise database or a collection of data marts managed by a DW group. Several authors highlighted integration-enabling technologies -- such as enterprise application integration (EAI), enterprise information integration (EII), service-oriented architecture (SOA), and Enterprise Service Bus (ESB) -- which make BI without DW not only possible, but relatively easy. Others focused on the business perspective of BI. They suggested choosing an architecture that best meets individual business requirements, supports the business drivers, and matches the readiness (or lack thereof) of the businesspeople to take BI enterprise-wide with a DW and the appropriate governance and commitment.

We begin with Mark Fung-A-Fat of the Massachusetts Medical Society, who regards BI with or without DW as an evolutionary process that many organizations go through. The first stage of this evolution is characterized by the Lone Ranger, a go-to person, either in IT or on the business side, to whom end users turn for their reporting needs. The Lone Ranger is a de facto BI analyst who pulls operational source data from here, there, and everywhere, giving the end users what they want when they want it. Sooner or later, the Lone Ranger becomes overwhelmed with too many and too complicated end-user requests, and those same end users begin to recognize the limited capacity of a Lone Ranger strategy. At this point, the organization evolves into the second stage of the BI evolution. A data-driven department in IT is chartered to consolidate operational source data into reporting databases and provide the end users with standardized data and reports, which are properly audited and secured. This works for a time, until IT becomes the bottleneck as end-user requests pour in faster than IT is able to fulfill them. This may lead to the third stage, the informed organization, which is often driven by a senior executive. In this phase, data integrity and business process management are critical to the perceived success of BI. As a result, data is cleansed, a mature methodology is used, databases are carefully architected to control redundancy and ensure consistency, a governance structure is put in place, and so on. Fung-A-Fat concludes his article by pointing out that all stages in the evolutionary process should be considered valid BI strategies.

In our next article, Bogumil Kaminski and Patryk Choros of Polish IT consultancy Infovide-Matrix (Cutter's partner in Eastern Europe) argue that BI with DW and BI without DW support two completely different types of end-user requirements, which they call end-user contracts. While some end-user contracts can best be supported by a single version of the truth, other contracts don't have that requirement. Instead, they may demand the capability to respond to change quickly, which might best be supported by an independent (silo) BI solution. As the authors note, "Development of BI solutions is always connected with a specific business case -- otherwise, no one would pay for it." End-user contracts for these business cases must take into account the variables of cost, time, quality, and availability. Kaminski and Choros give two examples of how to use cost, time, quality, and availability to choose the best BI solution architecture. While they do not write about an evolutionary process per se, they do describe various situations that require original end-user contracts to be reevaluated. This reevaluation can lead to a change in strategy from BI without DW to BI with DW. However, in order to be able to effect a change in strategy, it is important to manage the end-user contracts centrally. The authors conclude their article with tips and best practices for designing BI solution architectures that can migrate easily from one strategy to the other.

Next, Scott Ambler et al. of IBM Rational concede that many organizations have succumbed to the challenges and disappointments of BI with DW, but the authors warn us not to throw the "EDW baby" out with the "bureaucratic bathwater." The bureaucratic bathwater refers to such things as long-running and rigid waterfall methodologies, lack of end-user involvement throughout the project, the assumption that requirements were articulated properly only to find out too late that they were not, and so on. Rather than letting end users collect their own data using spreadsheets or even BI tools to produce their own reports, which would be throwing out the "EDW baby," the authors recommend changing the method of development and end-user interaction. Readers familiar with agile techniques will probably find it refreshing to see them applied to BI and DW projects. I hope that readers not familiar with agile techniques will extract much food for thought from this article, such as discovering how to apply the Rational Unified Process (RUP) -- or its open source or commercial equivalent --to deliver quickly by breaking requirements into use cases; how to involve end users in prototyping activities; and how to allow end users to refine their requirements during those activities. The authors also describe the concepts of agile data modeling, database refactoring, and continuous database integration, just to name a few. The article concludes with a list of best practices for adopting a lean data governance strategy to support all of the previously discussed agile techniques.

Taking an opposing view in the fourth article, Sumeet Mahapatra of Thoughtworks presents his arguments for how the future of BI will depend on business drivers as well as the direction the technology is taking. He writes, "In a world turning to service-oriented architecture (SOA), the question is whether massive data consolidation and data integration exercises are worth the huge effort." We already see forms of BI without DW, such as corporate performance management (CPM) and business activity monitoring (BAM), which are taking advantage of integration-enabling technologies. The trend appears to be continuing in that direction. Mahapatra's observations are not just based on technology trends, but on the changing demands of the business community. Business drivers that are inviting a BI-without-DW strategy include: the desire for lower infrastructure costs, faster and cheaper development, real-time monitoring, and improved data access and security. Furthermore, he notes that end users today demand operational real-time BI with browser-based BI applications, portability to handhelds, instant alerts, and continuous feedback on operational processes. He doubts that the traditional enterprise DW will be relevant in addressing these new end-user needs. Adding even more reasons for BI without DW, Mahapatra describes the most recent BI strategy of "analytics outsourcing," where the outsourcer collects the data from an organization, analyzes the data, and provides key analytical insights back to the customer. After citing a few more future trends that support his position, he concludes the article with the suggestion that a concept like Salesforce.com for BI is totally within reach and would probably render the "with-or-without-DW" debate moot.

Following the thread of the technology trends from the previous article, S. Radhakrishnan of Asian Development Bank compares the traditional BI-with-DW architectures to the new BI-without-DW architectures. While conceding that there are instances of large-scale enterprise-wide initiatives that still require traditional architectures based on extract-transform-load (ETL), data marts, DW, and operational data stores, the author asserts that in many cases the new architectures (based on SOA, EII, EAI, ESB, portal, and Web 2.0 technologies) can accomplish the same goals of those initiatives and more. Radhakrishnan provides a detailed accounting of the capabilities of these integration-enabling technology components and offers ideas on how to use them. For example, he illustrates how EII can be used to federate across popular relational databases and file dumps, and how EII opens up possibilities for "mashing up" data in unconventional ways. Readers of a technical bent will appreciate Radhakrishnan's comprehensive explanation of how an ESB backbone can be used as an integration platform not only for application integration, but also for data integration. The in-depth review of how to use the capabilities of the integration-enabling technologies on BI projects continues, covering portals, rich Internet applications (RIAs), composite applications, dedicated clients, Web clients, desktop widgets, and application widgets. As enticing and exciting as the new technologies are, the author admits that they pose a few critical challenges in the areas of metadata management, data quality, integration, and security, for which he offers a few good practices.

When you get to the last article, you may think that you have read every perspective on the topic of "BI with or without DW" there could possibly be, but don't stop now. Be sure to read Cutter Senior Consultant Bob Benson's passionate article "Who Cares? A Realistic Business Perspective on the BI/DW Decision." In it, Benson challenges the entire premise for a technical discussion of this topic. In fact, he states emphatically that the with-or-without-DW question is irrelevant from a business perspective. Businesspeople consider that merely a technical implementation matter. Having said this, the author quickly points out that he is in the BI-with-DW camp and that, in his mind, "the long-term implications for the business of data chaos and `spider web' operational implementations trump the (also) important short-term individual business unit (BU) issues." However, he insists that as long as IT folks couch their case in technical terms, the benefits of one BI strategy over another will remain obscure to the management team. Since the BI/DW decision must be based on business issues, Benson offers five key questions to ask before choosing a BI strategy. These are basically governance and planning questions that lead to two perspectives: a strategic perspective and a governance perspective. Using the five key questions, Benson compares two types of organizations: one with a sound strategic foundation and the other with a weak strategic foundation. For each type of organization, he suggests a course of action. Similarly, he applies the five key questions to the governance perspective, this time taking a quadrant-analysis approach to choosing the most appropriate BI strategy. He concludes his article by stating that in his view, "It doesn't matter how strong the arguments are for either position, or how well they are made.... The correct answer depends on the business."

I hope that the diverse opinions and experiences of our authors in this issue will give you the insight you need to negotiate the most appropriate BI strategy for your organization at this time. Remember that what's right for your organization today may not be right for it tomorrow. As our authors point out, BI is a journey -- not a destination.

ABOUT THE AUTHOR