Design Constraints and Unavoidable RealitiesNo design problem in school was this hardContinued from Page 1 Decentralized, incremental development. Very few of us are wise enough and staffed sufficiently enough to complete a perfect design of a data warehouse up front from a centralized position. The rest of us need to rely on designs developed in separate departments and done in an incremental manner as we learn the requirements of the users and the implications of the data. Our data warehouse must be a profoundly decentralized system, in some cases with no physical center at all. I can't resist describing my favorite centralized system metaphor. If the telephone system in the United States had been designed as a fully centralized system, then there would be a single massive switching center located somewhere in Iowa, and every telephone in a separate wire running to this location! The telephone system is a great example of a distributed system, and it's thousands of times more complex than any data warehouse. Multiple, incompatible technologies. Most organizations have a variety of operational systems, database engines, and end-user analysis and delivery technologies that, below a certain physical level, are profoundly incompatible. To integrate these systems, we need a sophisticated approach to applications and data communications at relatively high levels, divorced from individual data formats. Rapid deployment. End users think that six weeks is "about right" for the deployment of a data warehouse. At the other end of the spectrum, you'd better deliver something useful within a year because if not, your budget will go away, your boss will go away, and maybe your company will go away. Continuous change. The business world continuously changes, making a joke out of long-range design assumptions. The only recourse is a kind of design that invites yet is resistant to change. What kind of design is that? We'll see that the secret is symmetry. Data marts. The term data mart is important because it represents the insistence that individual groups source and publish their own data to meet urgent local needs. The real problem with data marts, of course, is that they're often stovepipes. As engineers, we eliminate the stovepipes and turn the natural energy of data mart development to our advantage. Data marts must become the central elements of our distributed data warehouse. Atomic, near real-time data. We need the most atomic data in our data warehouse in order to ask the most precise questions and drill down to the most operational perspective. We also need this atomic data to be as real-time as possible. Real time now is generally understood to mean less than an hour old, although it can often mean virtually zero latency. The data warehouse needs to track the operational systems in lockstep. Seamless history. As if the previous requirements weren't bad enough, now the data warehouse must connect the real-time data seamlessly to the static historical data to present at least the illusion of a seamless connection from the beginning of time to the present. Picking Ourselves Up Off the FloorIf you're still with me, stay tuned. In my next column, I'll dig us out of this seemingly impossible set of design constraints and unavoidable realities. We'll separate our physical systems so they perform focused tasks much more efficiently, and we'll use dimensional modeling to attack all these challenges in a predictable, re-usable, cost-effective way. Ralph Kimball [www.ralphkimball.com] co-invented the Star Workstation at Xerox and founded Red Brick Systems. He has three best-selling data warehousing books in print, including The Data Warehouse Toolkit, Second Edition (Wiley, 2002). He teaches dimensional data warehouse design through Kimball University and critically reviews large data warehouse projects. RESOURCESRelated Article at IntelligentEnterprise.com: "An Engineer's View," July 26, 2002:
|
 |
Most Popular This Week
IE Weekly Newsletter
Subscribe to the newsletter
| |
|