The data warehousing/business intelligence (DW/BI) industry is largely self-evaluating — those with the most to gain from favorable reviews are usually also in control of reporting results. Vendors naturally desire to emphasize the most positive and dramatic outcomes, organizations that spend large sums of money are reluctant to publicize failing or underperforming implementations, and analysts walk a fine line between informing their corporate customers and not provoking their vendor patrons. There are some fiercely independent sources of information, such as Nigel Pendse's OLAP Report, but sadly, it only covers a fragment of the market.

In the recent past, some analysts issued reports claiming that extract, transform, and load (ETL) was far superior to alternatives, but ETL vendors sponsored the study. Another study evaluated ROI and TCO for various ETL vendors and, not surprisingly, the ones that demonstrated the best results were also clients of the analyst firm. Many studies employ a shaky methodology, such as the famous Data Warehouse ROI report by IDC (1996 Foundations of Wisdom study), that found an average ROI of 401 percent, but a closer examination revealed that those with ROIs less than zero were excluded from the calculation. A more recent study by TDWI, one of the leading data warehouse education and research organizations, disclosed that almost 70 percent of its survey participants were IT, vendors, or consultants, not stakeholders in organizations who ultimately benefit from these initiatives.

As a DW/BI practitioner, I found these findings very suspicious as they didn't track with my experiences at Hired Brains: They were simply too optimistic and, in some cases, seemed contrived. Despite our best efforts, projects all too often fail to reach their goals for reasons that may not be obvious. It's easy to pinpoint the usual suspects, such as mid-project organization realignment and killer politics, our inability as consultants to convince our clients that certain decisions are suboptimal and a host of others, well documented in the literature (such as TDWI's "Ten Mistakes to Avoid" series). But there are also many cases where everything goes well, yet the initiative never gets traction in the organization, penetration stays at a very low level, and ROI projections aren't met. In many of those cases, this failure to thrive is something of mystery.

To understand how to help our clients achieve success with this technology, we at Hired Brains decided to go to the source and survey the actual BI stakeholders and try to understand the phenomenology of getting BI to work. What we found was very revealing (see Figure 1).

Basically, as an industry, we've mastered the technology, but we've failed at a cognitive level. We haven't been able to change the way people use information, particularly their reliance on standalone spreadsheets and personal databases. (See the sidebar, "In the Shadows.") This single finding not only explains the low penetration rate of BI but also, more ominously, torpedoes any hope of a good ROI, as people only have so much time in a day. If the time spent with desktop tools doesn't decrease, then the benefits from the BI effort won't be realized. We also failed to provide tools that are indispensable or even very useful. The relevance, integration, understanding, and workflow elements of our offerings are all sorely lacking. How can this be?

The Training Mindset

There are a few clues in the survey data, although they're only that; more in-depth research is needed and ongoing. Training clearly has a direct relationship to the perception of success. As the chart shows (see "Training Spend Vs. Outcome" in Figure 1), organizations that devote at least 16 percent of the project budget on training have a greater than 90 percent favorable rating for the data warehouse effort. Organizations that spend 5 percent or less, a pretty typical investment, have a largely unfavorable rating.

FIGURE 1 Data Warehouse dashboard from the user's point of view

Clearly, just spending money on training is insufficient for achieving success, but those organizations that invest in training expose a mindset — cooperation between technology and people and taking an interest in the work that people do — that affects all their decision-making. In those cases where training is given only a few dollars as an afterthought, stakeholders largely find the whole initiative irrelevant. It isn't the training itself that counts, it's what it denotes: An organization that takes an interest in the work that people do and forms a partnership to continuously improve the tools and the output by weaving technology into the work environment, not imposing it.