Intelligent Enterprise

How do you make "real-time" decisioning a reality? With data volumes growing and batch processing windows getting ever smaller, conventional data integration methods often can't keep up. Stephen Brobst, chief technology officer at Teradata, and Alok Pareek, vice president of technology at GoldenGate Software, make the case for change data capture (CDC) technology, process change and, most importantly, visionary business leadership to take advantage of real-time information.

An IDC survey recently found that more than 70 percent of respondents say they expect to use real-time data in their BI environments within the next 12 months, yet only 30 percent say they are doing so today. How do you explain that gap?

Stephen Brobst

Stephen Brobst: It really depends on the industry. Big retail in North America is already there, but in other industries there's less competition and less motivation to be innovative. Adoption is very much based on where there's competition. There are also plenty of companies that are moving toward using real-time information, but the hard part isn't the technology so much as the business process change.

Alok Pareek

Alok Pareek: In addition to competitive pressures, the volumes of data that companies are now dealing with are forcing them to rethink traditional batch processing. Many systems have been running in nightly batch processing mode for decades, but companies are now facing pressure to reduce data latency. The trouble is that if you try to stick with conventional extraction and feed your warehouse on an intra-day basis, it can have a big [negative] impact on your production systems. If you take a change-data-capture (CDC) approach, on the other hand, and you're doing data acquisition on a continuous basis, you can reduce the overall impact on your systems while reducing data latency.

What kind of "impacts" are you talking about and why is CDC faster?

Brobst: CDC uses log sniffing to retrieve only the data that has changed, rather than scanning and replacing big fat tables with lots of data. When you look at data stores getting bigger and nightly batch windows getting smaller, the equation for conventional data extracts just isn't working any more, in many cases, because of the impact it has on mission-critical OLTP systems. Batch extracts are resource intensive in terms of CPU and Input/Output cycles, and there is now precious little time to reach in, grab the data and do all that processing. With CDC, you're doing log sniffing much more efficiently to get only the data that has changed. You don't have to scan all the data, so the CPU and I/O impact is much smaller.