Intelligent Enterprise

There's been a blurring of the lines between data integration in the BI environment and in the operational environment. It used to be that you bought ETL tools for the DW, and mostly hand-coded data integration for OLTP projects.

The problems come when vendors obscure the differences between data and application integration to broaden the appeal of their tools. You'll find EAI and messaging vendors tout their tools for DI, and ETL vendors talk about operational DI.

When evaluating tools, it's important that you realize that data integration and application integration are not the same thing.

The biggest patent lawsuit ever to hit the ETL market is probably one you never heard of: JuxtaComm versus virtually every company in the market – a total of 16 defendants at the time. Teilhold Technologies, a subsidiary of JuxtaComm, filed the patent infringement claim late last year in the Texas Eastern District Court, a favorite of patent trolls because the courts there favor patent trolls.

The lawsuit is based on a 1998 patent (6,195,662) fpr ETL:

This video will help you understand the essence of the end-user experience for business intelligence, and the message is delivered in ten seconds. Keep it in mind as you deploy BI more broadly.

I was sorry to see last week's Teradata Partners user conference come to a close. The event was half Teradata technical sessions and half business sessions, where I spent more of my time. The business sessions were rougly equal parts enterprise BI platform and methods, data management topics, customer and business process BI, and active data warehousing. There were other topics mixed in as well, including a number of sessions that highlight the deepening company relationship with SAS.

While not a stated theme, I noticed an increase in Web data as the subject of analysis, or married to internal transactional and customer data. There was even a session on analyzing social networks based on cellular call data — not exactly Web, yet a topic most commonly associated with Web businesses. There were presentations by eBay, PayPal, and Netflix as well as non-Web companies and government departments.

Teradata introduced the Teradata 1550 "extreme data appliance" at its user conference this week. The appliance starts at 50 TB (based on compression) for a single node and can scale to 50 PB (theoretical data size). This appliance is positioned to deal with the very large data volume problem, not so much for typical data warehouse usage.

When you look at data usage, there are two types of large data problems. The classic DW model involves analyzing subsets of the total data, and occasionally scanning all the data. The other model is the need to analyze very large data sets that would normally be impractical, like looking at a year of web traffic or call details.

I'm at the Teradata Partners conference this week. I consider it to be the best event in the BI market if you want to see a diversity of company presentations, particularly on more advanced topics. You won't find the same number and quality of end-user presentations at any other event. The official kickoff went through some interesting and entertaining moments and closed with a terrific keynote from Dan Ariely, author of Predictably Irrational: The Hidden Forces That Shape Our Decisions. Here's my quick takes on each of the talks:

The people who say that open source has no impact or visibility in the data warehouse market were shown to be wrong at the TDWI conference in San Diego. We saw a continued rise in open source data warehousing products this summer. Jaspersoft, Talend, Ingres and newcomer Kickfire all had booths at this event. That's a big change from no presence roughly 18 months ago.

What's notable about this is that we aren't talking about just BI tools. This combination of vendors provides complete coverage of the standard data warehouse technology stack, from platform to BI. If you wanted to, you could acquire a complete open source BI solution at this event.

Talend announced an open source data quality offering today. The company has been moving very quickly to fill out all of the basic components needed for a complete data integration suite. In June the company delivered Talend Open Profiler for data profiling, and the Open Studio ETL tool contains changed data capture (CDC) features. While not a full suite yet, these are a big expansion of functionality in short time. Unfortunately you'll have to wait a few weeks. Data Quality won't be ready for download until September.

By acquiring DatAllegro, Microsoft is filling a performance and scalability gap that has kept them from consideration in larger data warehouse deals. Microsoft announced the acquisition today but has not yet disclosed the terms of the deal. DatAllegro just completed a series D funding round of $19.6 million in May, bringing the total funding over their five years of existence to roughly $63 million.

DatAllegro has been secretive over the past few years about its customer base, leading some analysts (including me) to wonder how well they're doing in the highly competitive data warehouse platform and appliance market. They have only three customers that I know of, but they say that the largest of these sites are storing hundreds of terabytes. This offers a compelling scalability story for Microsoft once the DatAllegro technology is merged into SQLserver.

I thought the billion-dollar FAST deal Microsoft made was crazy based on my conversations last year with FAST about their products and prospects. The Microsoft presentation at the Independent Analyst Platform in Phoenix last week reminded me to follow up on things that have been sitting in the queue for a couple months.

In the world of Business Intelligence, Excel is the devil and BI tools are the savoir. Spreadsheets are a satanic element we're trying to drive from unrepentant departments. This is because centralized data is good and distributed data is bad.

Ideologies exist to simplify the world by trying to separate everything into two buckets: good and evil. A spreadsheet being used for BI, like the devil, is the embodiment of badness. As with all good demons, a spreadsheet entices people to make bad choices and do bad things like retaining data in files on their PCs.

How does it entice people? By offering something of value, like simplicity or ease of use. Ever try to add a quick formula to a Web-delivered report? It can be a nightmare. Export the report into Excel and it’s a snap. A spreadsheet is malleable, unlike the rigid offerings of the BI orthodoxy.

James Dixon had a comment on my services/cloud post worth exploring as it's about a fundamental criticism that's been around since the first ASP started years ago:

"Doesn't DW-in-the-cloud suffer from the same fundamental problem as DW-as-a-Service in that you have to pump all of your proprietary, strategic, highly sensitive data outside of the firewall onto someone else's hardware?"

I think that's a valid argument, provided your company has no external network connectivity. If you have an external connection, then all bets are off. It's worth looking at some networking pre-history to see why this has been true for decades.

I've been getting caught up on briefing notes and press releases since the TDWI conference and I've come to the conclusion that "as a service" is getting played out as a marketing term. The new and exciting term now being borrowed from the Web world is "in the cloud."

Companies have been throwing out the " aaS" as in SaaS in favor of " itC." While there is a difference between the two, many companies never figured out if they were SaaS or a managed hosting environment. Now they're doing the same thing with SitC and SaaS. I expect to see more confusing messages in the market as vendors rush to the next buzzword.

I heard about an interesting presentation planned at the EMC World conference in May. According to this talk, EMC will introduce another entry into the warehouse appliance market via a partnership, this time with ParAccel:

"EMC and ParAccel have jointly engineered and developed a highly scalable and performant analytic appliance. This solution is built on EMC CLARiiON midrange CX-3 UltraScale networked storage and ParAccel's analytic columnar data store. Customers can deploy the EMC/ParAccel analytic appliance by simply extending their existing EMC footprint on enterprise ready storage while leveraging EMC's proven solutions."

Microsoft's bid for Yahoo is certain to shake up the online advertising, Web 2.0 and open-source markets. Yahoo has been on the path to being a real player in the Web and open source world. They've released tons of code via their developer programs and pushed some really innovative services aimed at Web developers. Being absorbed into Microsoft could actually hurt the industry in these areas. I'd expect less open source support out of them for a start.

There's definitely going to be cost-cutting after the deal is done. There are some specific quotes from Microsoft's offer worth highlighting.

The reasonable-sounding CRM conventional wisdom is that you should "fire your low-value customers," but it turns out to be not so reasonable after all. The theory is that low (or negative) value customers are a drain on limited resources, so getting rid of them should raise margins and make the company more profitable. Except it doesn't, according to a recent study by two Wharton marketing professors.

ParAccel announced top TPC-H benchmark performance numbers with Sun at the end of October, beating out the former leaders in both the price and price-performance. Not by a little, but by four times in performance with a big drop in cost. I haven't seen much discussion of these results.

The fact that a little startup like ParAccel can enter the market with a database to support business intelligence that beats the TPC-H results of all the major vendors on both performance and price should wake people up. Particularly when the performance increase is so large while significantly decreasing cost.

People are acting surprised because IBM bought the last of the big BI platform vendors. I can't figure out why. The marriage of IBM and Cognos has been whispered about for several years. IBM kept saying "we don't want to be in the application business," but they also weren't in the ETL business or the content applications business either.

As a software vendor, IBM is all about enterprise infrastructure. BI as it's talked about today is mostly reporting infrastructure, and as such has become mainstream infrastructure — exactly the sort of thing IBM does.

Last week I attended the Teradata partners conference, one of the best events to go to if you want to see some of the leading-edge things people are doing. Unlike many conferences, this one has a lot of case studies, and they set a high bar for quality. Since Teradata sits at the core of the warehouse, they get a broader range of speakers, so I always find something of interest.

SAP and Business Objects are calling the acquisition of the latter a friendly takeover, although I wonder how many employees will view it that way. With this purchase, the BI market now looks a lot like the ETL market. There's only one large independent vendor in each market - Informatica for ETL and Cognos for BI - and a bunch of mostly smaller companies left.

This is a great thing for SAP since they can now start taking in sales for BI where once it all went to third parties. They also get a decent set of data integration and data quality products to complement SAP's sore applications. It's good for Business Objects too, since this opens up the market for all those SAP accounts.

Web 2.0 technologies are going to change BI, possibly undercutting demand for conventional BI software. People wonder why I keep saying this. Here's a great example: real estate search.

Not that long ago, you would look at listings in your price range and try to work out where they were and whether they were in nice neighborhoods. More likely, a real estate agent would do this for you. Their value was almost entirely access to information and knowing how to get at it via the multi-list service.

None of the shopping sites with which I'm familiar truly take advantage of the presentation opportunities offered by the Web. They still merchandise online in the same way they lay out shelves in a store. So you filter by type of clothing, style, gender and size.

Speaking at the recent TDWI Executive Summit, I talked about Web mashups, BI and the blurring between the two. One of the mashups I showed was ColorPickr, a nifty app that pulls images from photo site Flickr based on your choice of color from a palette.

Every TDWI conference leaves me with insight into what to expect during the next year. At last month's World Conference in San Diego we had both the regular conference and a two-day executive summit. The main trends I saw:

Predictive Analytics still hot — Predictive analytics (a.k.a. data mining) seems to be the topic most people are interested in hearing about. A lot of the engineering problems we faced with data mining in the '90s have been solved and cheap computing power makes broader use more feasible. The catch is that it still takes expertise to understand which techniques work best for different problems. Expect a new subclass of BI professionals who know PA tools and techniques, just as we have BI tool and design experts now.

Lots of people talking about data governance — There seem to be two threads driving this: