I’m actually currently running a survey on the topic. I'd very much welcome your participation. The survey is titled "Continuous Intelligence: BI Beyond Real Time." You can respond anonymously. The aim is to understand the needs of data and analytics users. The survey sponsor, Aleri, will send the first 200 respondents a $5 Amazon gift card in thanks.

I realized only yesterday that Aleri has trademarked the "continuous intelligence" term. That's disappointing. While there are definite advantages to owning a distinctive, well-chosen term -- independent of Aleri's sponsorship of my work, I think "continuous intelligence" is a very apt term for BI-style application of Complex Event Processing (CEP) tools and techniques applied EP for BI," -- I think that Aleri would be better served by broad adoption of the term.

By contrast, Forrester Research analyst James Kobielus has put forward the term "really urgent analytics," which he characterizes at "the sweet spot for real-time data warehousing." But that term relays only half the picture. It doesn't (fully) acknowledge the data side, that users need real-time data and not just fast analytics. More recently, on twitter, Jim has advocated simply "eventing," which to me is vague. Jim plans Q3 research on the topic. He's very astute, so I'm sure he'll be refining his thoughts and that his report will be very worth reading.

The idea behind continuous intelligence and other adaptations of CEP to BI contexts is to drive faster decisions and actions across the enterprise with real-time analytics on both real-time and historical data, the latter typically managed in a data warehouse. (I arrived at some of this language in the course of discussions with Aleri marketing VP John Morrell.) CEP software is designed to analyze and transform "data in flight" without requiring data to be loaded to a data warehouse. It additionally supports sliding time and record windows -- for example, all data received in the last 15 seconds or the last 20 records received -- with long-running queries registered in the server to wait for and act on the occurrence of data events.

By the way, my definition of "event" is "the change in state of an observed or derived (computed) or implied variable of interest." Key performance indicators (KPIs) are one class of derived/computed variable. Quantities such as stock-market indexes are another. And by an "implied" variable, I mean one that is neither directly measured nor computed. We get back to the old conundrum "if a tree falls in the forest and there's nobody around to hear it, does it make a sound?" That question sounds a bit dumb nowadays, but it helped philosophers such as Bishop Berkeley define notions of existence and perspective and experience.

The ORCL (Oracle) share crossing the fifty-day moving average price is an example of a data event. (It's a "data event" because it's artificial, not associated with any physical-world action.)

You can do event processing off a conventional data warehouse if you can tolerate the latency (delay) involved in data capture, structuring, and (often) indexing. One point that Dan Graham, senior marketing manager responsible for the Active Data Warehouse program at Teradata, made in the course of a Teradata-sponsored webinar I recently gave on Event Driven Analytics is that analytical data warehouses such as Teradata, which reduce data latency to as little as 30 seconds, handle the vast majority of event-driven BI needs. This is an important point and has surely fueled the lack of perception of need for faster, BI-oriented CEP-style event processing offered by companies such as Aleri, IBM, Progress, Streambase, TIBCO, and Truviso.

Here I'll interject that Richard Tibbetts, CTO at StreamBase, says that "event processing is an application style," and he's completely right. It's a big tent that covers a range from sub-second (even millisecond) CEP response to the within-minutes response supported active DW vendors. Analyst Brenda Michelson, again on twitter, says "an event is a thing that happened; event processing discerns the notability of that thing, & if applicable invokes action." (Brenda published a clear and useful Event-Driven Architecture Overview back in 2006 that is still worth reading.) And the estimable Neil Raden has been nosing around the topic, in connection to (or perhaps distinction from) his interest in process intelligence and enterprise decision management.

So the rest of 2009 and through 2010 could be when low-latency event processing, whatever you choose to call it, comes of age as a BI option and asset. I’m not placing any bets, but I will again plug my survey as a way you can help me find out.

E-MAIL  |