Intelligent Enterprise

It's bad enough dealing with information overload — we shouldn't have to deal with a glut of tools to find the right information as well. Yet that's the position many organizations find themselves in today.

Businesses typically use anywhere from four to eight different search engines, according to Delphi Group. And while information assets include both content and data, the tools available historically haven't brought those two sources together; search engines have targeted documents and other forms of content while business intelligence tools have queried data.

Frustrated searchers, take heart: Consolidation is the trend in search, and it's beginning to help organizations achieve cohesive information intelligence. This trend is illustrated by two main developments. First, the data query and content search worlds are converging to give businesses a single point of access to structured, semi-structured and unstructured information. And second, organizations are attempting to resolve the "search-engine glut" problem either by consolidating results through federated access to existing search applications or by providing a single search solution across the enterprise.

The Urge to Converge

Content and data are two valuable information sources that have been managed in very different ways. Content is rich in context but lacks the structure needed to do the slice-and-dice, iterative analyses possible with BI tools. Data has structure but its context is the database: Searching a database and extracting fields gives users little meaning about information.

Bridging these two worlds has been a long-standing objective in the vendor community. "It's where the whole search market is heading," says research vice president of content technologies at IDC Susan Feldman. "After all, it's all just information in the end."

While still in the early adopter phase, the convergence of data and content is being realized in real-world deployments. Converged applications allow users to conduct searches — or, better yet, ask natural-language questions — that combine both descriptive text searches and database queries. Feldman cites an e-commerce scenario in which a consumer might ask, "Which blue cotton sweaters [a search of text] sell for less than $30 [a database query]?"

Supporting this type of inquiry is the goal of Chelsea & Scott, a retailer of children's products under the OneStepAhead and Leaps and Bounds lines. A keyword-based search feature on the company's Web site didn't address the fact that customers often don't know what products to search for, according to Rachel Pendon, e-commerce manager. More often, customers are looking for solutions to problems such as "helping children sleep better at night," Pendon says. However, that type of "purpose" information wasn't a category in the product database that fed the company's Web site.

In May, systems integrator Tellus helped Chelsea & Scott deploy the OneStep natural-language "adaptive search" solution from iPhrase. As part of that effort, the company identified new attributes such as "bedtime comforts" that were neither data fields nor textual product descriptions. The search solution can now suggest products with similar attributes even if they're not in the same category in the database. The technology can also overcome classification errors. For example, it can appropriately list a product described in text as "great for children under three" even if a numeric age range wasn't specified in the database.

Among the vendors pursuing convergence trend is Fast Search and Transfer (FAST), which in early 2004 released the internally developed FAST ESP platform, which searches data and content. Text analytics vendor ClearForest and search vendors Endeca and iPhrase have also developed convergent capabilities internally. In contrast, search vendor Autonomy added converged content and data search capabilities earlier this year by acquiring Ncorp, a company that specialized in technology for querying structured data.

Vendor approaches to convergence lean toward the sector — data or content — in which they started. Data-centric approaches use techniques such as text mining to turn content into data, thereby creating another source to query. Content-centric approaches pull text from databases into an index that gives it context based on the original location of the records.

What's most important isn't the method, but the effectiveness of creating a single point of access that will support better decisions by finding relationships in information that couldn't have been easily associated without these techniques. Energy giant BHP Billiton (BHP), for example, needs to make the right decision about whether or not to begin develop new petroleum fields. The company typically spends anywhere from $20 to $100 million just on exploration and discovery of a single new well and another $1 billion on a field if it decides to drill and develop.

BHP stores documents on current and past projects in the central repository of its content management system (CMS), but it found that its metadata-based search engine didn't discover all significant documents related to a particular project or line of inquiry (in part because authors hadn't been consistent in applying metadata). Nor did the engine take into account structured data available in well reports and geologic databases.

"The problem is that in certain cases a lot of the information may lie in a database, but in other cases, it may just be a graph in a PowerPoint presentation," says Michael Glinsky, section leader in quantitative interpretation at BHP and the leader of the company's recent search project.

In early 2005, integrator Blue Fish helped BHP deploy Endeca's ProFind search tool on the company's intranet. Rather than relying on existing taxonomy or data structure, which can limit possible paths to records related to a particular search, ProFind explores every valid pathway to results and then presents "guided search" contextual and classification information alongside the conventional hit list. For example, the system can break out separate pathways just to the articles, photographs, tables, audio or maps related to a search. Glinksy says that this allows BHP engineers to quickly refine searches and drill down into smaller, more relevant collections rather than browsing through long lists of unrefined results. The approach also helps engineers working on new projects to discover similar projects and proven solutions.

BHP's new system isn't passive: When new projects are launched and related documents are checked into the CMS, Endeca automatically searches for projects with similar characteristics and pushes that information to engineers. BHP spent about $1 million on the project, an amount the company expects to recover quickly because the system will enable engineers to identify more accurately oil fields with the greatest potential for successful production.

"We had a mountain of information and no effective way to go through it," Glinsky says. "Now we're getting better value out of millions of dollars of research and influencing decisions that can create hundreds of millions of dollars of revenue."

To Federate or Consolidate?

Search engines have multiplied within many companies. They show up on Web sites and in content management systems, as appliances used to search intranets and as browser plug-ins employees use to explore their hard drives.

"We've developed over time this legacy of search being delivered within many enterprise applications," says Hadley Reynolds, Delphi Group vice president and director of research. That legacy is unlikely to end any time soon, with search vendors continuing to partner with business application developers. For instance, Siebel announced in April that it will embed the FAST search engine into its CRM platform.