Intelligent Enterprise

Google keeps raising search expectations, announcing in December, for example, that it will scan and full-text index books from the libraries of Harvard, Stanford, Oxford, the University of Michigan and the New York Public Library. Publishers of proprietary information, meanwhile, decry what Amanda Spiteri calls the "Googlization" of research. "There's a growing mentality, particularly among the younger generation, that if it doesn't come up in an Internet search engine then it doesn't exist," says Spiteri, marketing director of Elsevier's ScienceDirect, a subscriber-based service for libraries, universities and research institutes.

To counter this attitude, Elsevier and other publishers are adding deeper information to their services. ScienceDirect has evolved from a Web database of Elsevier scientific journals and now links to 170 other publishers and a total of 1,800 scientific, technical and medical journals.

In 2001, Elsevier launched Scirus.com, a free search engine that indexes 167 million science-related Web pages, including millions of .edu, .org and .gov sites including Elsevier and other proprietary sources. And in the past year, Scirus has focused on indexing more top-quality science sources, such as the journals of the American Institute of Physics.

"It's often difficult for users of public search engines to distinguish good information from bad," says Susan Vugts, marketing manager for Scirus. "Our crawlers look specifically at scientific sources, and we developed our own scientific taxonomy that improves the accuracy of indexing and search results."

At Thomson Finanical, a supplier of real-time corporate data to the financial services industry, Internet competition from sources such as the SEC's Edgar database have led the company to add more detail. Thomson republishes all Edgar data, including XML, HTML and JPG attachments, as soon as it's available, but it also converts that data into ASCII and adds refined indexing.

"Edgar content has some tags, but they're not consistent, and they don't give our customers what they want from income statements and other documents," says Mary Ann Wismer, Thomson Financial director of document management. "Indexing is critical in terms of timeliness and making sure that facts are findable, so we add 15 to 20 index values to every document type. "