|
||||||||||||||||||||||
The market for data warehouses and data marts as autonomous, bundled sets of technologies has evolved and matured. No longer limited to early adopters, they have been deployed by many companies across the technology-adoption curve. Regardless of the success or failure of such efforts, the goals of data management architecture are expanding to encompass more than simply the construction of consolidated data sources. These goals now include the broader mission of making the greatest possible use of any and all available data assets.
A superficial assessment might dismiss data warehousing to a merely supporting role in the new “digital” economy, perhaps as an enabler for the delivery of decision-support data over a corporate intranet. But many companies, most notably in the financial services and retail industries, are beginning to expand this role by actively applying data warehousing techniques and technologies to leverage data resources across the multiple channels of the digital economy.
Big, Fast, and Channel-Neutral
Competition will be fierce and fast-paced in the digital economy. As one writer put it, “Failure will come for the unwitting just as quickly as success has come for the fortunate.” In this case, the “unwitting” would be those who do not effectively use their resources, such as their data, to constantly optimize the value chain: Margins will be razor-thin, and you’ll have to constantly monitor and adjust the costs and profitability of products, customers, and multiple supply and delivery channels.
FIGURE 1 A multichannel business model.
To compete in this digital economy, a business must establish and maintain a presence of substance and agility across multiple channels — connections — within its marketplace. (See Figure 1, page 46.) Initially, companies established this presence with “bricks and mortar.” Eventually, they extended the interactions forming their supply-and-delivery mechanisms into analog electronic channels: telephone call centers, interactive voice response, computer-telephony integration, and dial-up lines. Now companies are adding the digital communication capabilities offered by the Internet — Web, email, chat, and voice over Internet protocol — on top of these established channels, all of which must be maintained, integrated, and actively managed. Furthermore, customers and business partners expect the ability to choose the communication mechanisms most convenient to them.
The Value Chain
A useful planning tool for guiding migration to a multichannel business model is the widely accepted concept of the value chain, introduced in the book Competitive Advantage: Creating and Sustaining Superior Performance (Free Press, 1985) by Michael Porter of the Harvard Business School. A value chain is a high-level model of how businesses receive raw materials as input, add value to the raw materials through various processes, and sell finished products to customers. (See Figure 2, page 46.)
FIGURE 2 A generic value chain.
A critical prerequisite for success in the digital economy is the implementation of an integrated value chain that extends across — and beyond — the enterprise. As Cambridge Technology Partners defines it in a 1998 white paper (“Competition’s New Battleground: The Integrated Value Chain”), value-chain integration is the process in which multiple enterprises within a shared market cooperatively plan, implement, and manage (electronically and physically) the flow of goods, services, and information from point of origin to point of consumption. Furthermore, they do so in a manner that increases customer-perceived value and optimizes the efficiency of the chain, creating competitive advantage for all stakeholders involved.
In this article, I’ll take a quick back-to-front tour through the value chain, looking briefly at the technology challenges the new digital economy poses within each link. Then I’ll discuss in more detail what data warehousing technologies can offer to address these challenges, with the goal of transforming a “predigital” value chain into an integrated value chain.
The Integrated Value Chain Explained
Let’s take a closer look at the value chain shown in Figure 2. The “inbound logistics” link on the left side represents the supply side of the enterprise. In the digital economy, your business must have the ability to exchange data with suppliers quickly and easily, regardless of format. Data formats can be based on standards, such as extensible markup language (XML) or electronic data interchange (EDI), or use ones proprietary to a group of partners. (Proprietary formats, for good or ill, can be effective barriers to entry into a particular trading market.) Thus, inbound logistics systems must recognize and understand data originating outside the enterprise, and also replicate and transform it for use in internal and external “downstream” processes.
In the center of the value chain are the operational activities where the “value add” occurs; they serve as the “back office” in which the PCs are assembled, the pizza is baked, or the stock trades are executed. The digital economy enables — as well as requires — all operational activities to share data at maximum network speed among internal and external partners executing the value-adding processes.
On the right side of the figure, outbound logistics, marketing and sales, and customer service and support are the customer-facing links of the integrated value chain. Customers and prospects require authorized read-and-update access to relevant enterprise data that supercedes obstacles presented by operational, internal application silos. Compounding this challenge, the digital economy has caused an explosion in the number and variety of delivery channels through which companies can interact with customers. Consequently, your company must consolidate, aggregate, and deliver data over the Web and any other outbound channels intuitively and immediately.
Now that we've completed our whirlwind tour of the value chain, let's look at how some venerable data warehousing technologies can address the challenges involved.
Data Warehousing Technologies: Key Enablers
We can roughly divide data warehousing technologies into input (movement, replication, and transformation), storage, and output (information dissemination and analytic applications) categories. These “conventional” data warehousing technologies can be glimpsed under the covers of the new paradigms and products that are proliferating with the advent of the digital marketplace. For example, “new” enterprise portal and enterprise application integration (EAI) technologies, while not exactly old wine in new bottles, are much indebted to precedents set by data warehousing.
We can unbundle and reapply these component building blocks to form and strengthen crucial supporting links in an integrated data architecture—as well as the channels through which the digital enterprise connects partners, suppliers, and customers. This architecture in turn enables the integrated value chain.
Data Movement, Replication, And Transformation
Data warehouses and data marts are usually classified as data consolidation architectures, where data from many sources is transformed and replicated into a single destination—a many-to-one model. Data warehousing consolidation typically occurs in a periodic, batch mode rather a than a synchronous, real-time basis.
An integrated value chain is likely to be supported by the very same multiple, autonomous transactional databases confronted by earlier data warehousing efforts. Transforming a predigital value chain into an integrated value chain requires data movement, transformation, and replication technologies that provide real-time or near-real-time synchronization of multiple sources with multiple targets—a real-time, many-to-many model. Table 1 compares and contrasts these two architectures.
| Property \ Architecture | Data warehouse architecture | Integrated value chain data architecture |
| Synchronization frequency | Periodic (daily, for example) | Real-time or near real-time |
| Source/destination model | Many-to-one | Many-to-many |
| Update mode | Batch | Transactional |
| TABLE 1 Data architecture comparison. | ||
The exploding market for EAI software is a response to this growing need for many-to-many, transactional, near-real-time synchronization among data stores. Most EAI products, such as ActiveWorks (Active Software Inc.), Mercator (TSI International Software Ltd.), and BusinessWare (Vitria Technology Inc.) bundle several functions and technologies, including message brokering, workflow automation, and, most familiar to data warehousing technologists, data transformation and replication.
EAI software products achieve such data transformation and replication through message-broker middleware. Say, for example, applications A, B, and C are found in various value links and channels across an enterprise. Assume also that an update to the database in source application A must be replicated near-real-time to “target” databases in applications B and C. EAI software enables A to transmit an update message containing the updated data to a central broker facility. The broker “hub” then performs any required transformations and routes update messages to applications B and C.
Building on this model, your organization can achieve a progressively higher level of integration across the enterprise by connecting data from additional applications through one or multiple hubs. Multiple hubs can themselves be connected, thus creating the underlying framework for an integrated data architecture, and consequently for the integrated value chain. Furthermore, as almost a side effect of its data messaging and transformation functions, this hub structure will necessarily become the keeper of a widespread, distributed repository of data definitions. Could this structure also form the basis of a long-awaited holy grail: an active, enterprisewide metadata repository? Data management professionals will do well to follow these developments with high interest.
Data movement between channels.
Currently, commercial software packages that support interactions with customers or business partners tend to be specific to, and optimized for, a single channel, such as the Web. The digital economy, however, requires that any given channel be aware of the constituents, products, and transactions not only within its own domain, but also the constituents, products, and transactions of any and all other channels through which the company interacts.
For example, if I were to call in a retail order or stock transaction over my cell phone, I will certainly expect to be able to access a record of that transaction through the vendor's Web site immediately afterward. In addition, on my browser page the transaction record should indicate the channel through which it was received, as well as triggering personalization of the page based on the type and size of the transaction, and any purchasing trend it may indicate. Given the current state of software technology, an integrated-value-chain data architecture, such as that enabled by EAI products, is likely to be required to make this scenario a reality.
Data movement between front and back office.
Bricks-and-mortar companies discovered early that entry into the digital marketplace is not a simple matter of bolting a Web page onto operational systems. Unless it has the luxury of being a startup, a company moving into the digital economy probably has a legacy application infrastructure already in place.
These heritage databases will not be replaced, but must be wired to, and synchronized with, the databases of multiple inbound and outbound channel applications. For example, data originating from inbound and customer-facing channels must synchronize with back-office operational data. Similarly, orders coming in over the Web must be matched with inventory records, perhaps back-ordered with suppliers, and then scheduled for delivery. Amazon.com is a prime example. It lets a customer not only establish an account and immediately begin placing orders, but also track the status of orders up through shipping.
A new Web-based business enterprise such as Amazon has the luxury of near freedom from any stovepipe order-entry, inventory, and shipping applications. Such a business, created for the digital economy, may be able to enter the market armed and ready with an integrated value chain. But on the downside, few of these new digital enterprises provide choices of channels beyond the Internet…yet.
In essence, the very same integrated-value-chain data architecture prescribed for combining the newer and older interaction channels of a newly “digitized” enterprise can also wire its front and back offices together.
Data Storage and Related Technologies
Data storage technologies are the “content engines” humming behind the colorful storefronts of the digital economy. From the outset, data warehousing solutions pushed the envelope on data storage technologies; whatever the challenge imposed by the digital economy, data warehousing has probably already been there and taken it on. For example, the most critical data storage challenges today are security, scalability, reliability, and availability. Data warehousing brings to the digital economy especially valuable lessons learned in assuring the scalability and security of large databases.
Large data warehouse environments, spanning multiple value-chain links and constituencies of varying technical sophistication, typically include robust security capabilities for defining users and user groups, as well as the scope of data access available to their various constituencies. Protecting and leveraging critical information assets in the digital enterprise requires extension of security functions to a large number of constituents outside the enterprise as well.
Increasingly dynamic and vibrant computer-human interactions will also demand constant enrichment of basic textual and numeric content. Inline storage of video, audio, and image data will create significant scalability hurdles. Searching, accessing, and delivering multiple media through digital channels—with predictably rapid response, of course—will challenge the talents and expertise of the most leading-edge and experienced data warehousing professionals. Above and beyond this challenge, databases powering the digital enterprise will require automatic scalability in order to absorb unexpected transaction volumes, in spikes and sustained onslaughts, without hiccups. Database products as varied as Microsoft SQL Server 7.0 and ObjectStore by Object Design Inc. provide unique and innovative solutions to the automatic scalability challenge.
Data warehousing has also already faced (and overcome) challenges in database reliability and availability, but the digital economy has raised the bar to zero tolerance in these areas. If an internal decision-support database is down for the afternoon, business will go on, but if a database that powers an e-commerce site is down for an hour, business, by definition, does not go on. Reliability also dictates predictably brisk response to queries requested by internal and external constituents. In most cases, this requirement necessitates separate but synchronized data stores, each optimized for either input or output. This approach, of course, is the familiar data warehousing architecture pairing operational data store and decision-support database.
A quantum leap in predictably brisk response to queries against high-volume, read-only data stores may be provided by the new in-memory databases from companies such as TimesTen Performance Software Inc. and Angara Database Systems Inc. As Hector Garcia-Molina explains in “Fastest in Memory” (Intelligent Enterprise, August 24, 1999), memory prices have declined to the extent that the cost of memory necessary to store a significant database whole is now feasible.
XML: Common Language Of the Digital Economy?
XML is already having a substantial impact on how the data about digital transactions is defined. Volumes have already been written about XML, but the most important point to consider here is that it lets the client download structured, formatted data where users can view and manipulate it locally with potentially nothing more than a browser, dramatically decreasing server loads in the process.
So where does data warehousing come in? In large part, XML is concerned with data definitions—metadata. The data specialists (data analysts, data modelers, and DBAs) with the training and experience in developing and maintaining definitions for widely shareable data need to familiarize themselves with the data definition language of the digital economy as soon as possible. If not, history may repeat itself, and this metadata definition process will default once again to programmers, whose intentions may be good but whose motivations are typically more local than global. Data warehouse professionals need to learn XML—especially document type definitions (DTDs) and XML schemas—and seize control of its data-definition potential.
Information Dissemination: Portals
One industry pundit recently observed that an enterprise information portal (EIP) looks very much like a data warehouse with a browser on the front end. This observation is certainly not far from the truth. The primary difference is that whereas a data warehouse is a single data source, an EIP allows multiple data sources, regardless of format (or lack thereof), to appear as a single source—essentially, serving as a “virtual” data warehouse.
The bottom line is that no internal or external constituent of a digital enterprise will long be content in a business-intelligence environment where the physical location or format of stored data is of any visibility or consequence whatsoever. There is no doubt, however, that removing these barriers remains a major challenge. Once again, data warehousing techniques and technicians have amassed a long history of transforming data from one format to another. XML may hold a good deal of potential for meeting these challenges as well, due to its ability to define, store, and present content in multiple, disparate formats.
Analytic Applications: Managing Performance
Managing corporate performance in a digital enterprise requires the consolidation and analysis of cost and revenue data from multiple channels and value-chain links, on a near-real-time or real-time basis. Analytic applications tap the data that accumulates during the course of doing business, add value by summarizing and analyzing it, then deliver the results through dissemination services such as EIPs.
Using analytic applications, your organization can continuously monitor the relative activity level and profitability of channels, customers, and products to ensure responsiveness to changing market and competitive conditions, as well as to control costs. The goal is to increase the profitability of the most active channels, and conversely, increase the activity of the most profitable channels.
Many inbound channels capture data on literally every telephone button-push or mouse-click in every interaction with every customer. You can analyze this data to determine, for example, how often customers decide to “zero out” to a human operator, rather than using menu options to finish their transaction. A high volume of zeroing-out indicates that menu options and structure may require changes, and that the per-transaction costs have escalated due to increased involvement of customer service representatives (CSRs). CSRs require salaries and benefits; computer-based interaction units are much less expensive. If electronic channels are made more effective, human interactions are decreased, costs are driven down, and profits rise.
Real-time data mining of channel and customer data can drive automatic interaction personalization such as customized content presentation and navigation, as well as tactful cross-sell suggestions. These activities are logical applications of the “actionable” output resulting from established decision-support techniques including data consolidation, data aggregation, statistical functions, and predictive modeling. Many software companies—such as Epiphany Inc., Broadbase Software Inc., and Personify Inc.—are concentrating on providing automated support for these analyses and their results.
FIGURE 3 Data architecture in the digital economy.
Figure 3 summarizes at a high level how data storage, analytic applications, and data integration technologies combine to create an integrated value chain: the foundation of a digital enterprise.
Putting It All Together
As you can see, regardless of the perhaps battle-worn reputation of data warehouses, data warehousing technologies are far from approaching retirement age. Even if operating undercover and sometimes under multiple aliases, data warehousing clearly has an essential role in powering the digital enterprise.
RESOURCES
Hay, David. “XML: What Is It, Anyway?” IntelligentEnterprise.com, www.intelligententerprise.com/db_area/archives/1999/990308/online1.jhtml |
William J. Lewis (datamodel@aol.com) is an associate director in the analytic business solutions practice of Cambridge Technology Partners. He has more than 20 years of experience in IT, and has contributed to Database Programming & Design, The Data Administration Newsletter, DM Direct, and IDUG Solutions Journal.
|
|
|
|
