Five Principles of Intelligent Content Management

As the amount of unstructured text out there steadily grows in size, intelligent information retrieval techniques are all the more important for keeping decision-makers on track

By Dan Sullivan

Enterprise information portals and other content management applications have many purposes, but most of them share a common goal: to give large numbers of knowledge workers access to large amounts of unstructured text.

Content management applications have become increasingly integrated and sophisticated, but nevertheless, we continue to suffer the fact that no single tool, algorithm, or technique can solve the problems posed by enormous volumes of information expressed in complex and ambiguous natural languages. The range of possible topics, our means of expressing them, and the difficulties in teasing out the structural relationships within natural language texts are so demanding that we are unlikely to find a "silver bullet" anytime soon.

In this article, I'll explain how "intelligent" content management techniques that explicitly represent and exploit implicit information and underlying relationships in unstructured text can help senior IT managers and business leaders give their users better control over the information retrieval process. We can classify these techniques into five design principles that serve the same objective: reduce the effort required to find relevant content such as marketing reports, customer communications, and news in ever-expanding repositories of text.

PRINCIPLE 1: MAKE METADATA KING

Metadata about content serves several functions. First and foremost, metadata describes the essential aspects of text, such as main topics, author, language, publication, and revision dates. This type of metadata is designed to improve the precision of full text and keyword searching by letting users specify additional document attributes. It is also helpful for classifying and routing content, purging expired texts, and determining the need for additional processing, such as translation.

Managing content metadata involves two main challenges: extracting it from text, and then storing it. The extraction process needs to detect information ranging from the author's name to the dominant themes in the text while the storage process must efficiently support a number of access and retrieval methods. Manual extraction of metadata is effective on a limited scale, but in general, automated methods such as Solutions-United's MetaMarker, Klarity's eponymous metadata extraction tool, and Inxight Software Inc.'s (a Xerox spin-off) Categorizer are more helpful. Automated methods will not produce the same quality of metadata as a human being would, but they are effective for most purposes.

Regardless of how metadata is generated, you can store it in two ways. First, you can manage it separately from documents in a relational database. This approach is typical in document warehousing, where tight integration with other database applications, such as data warehouses, is required.

Alternatively, you can store the metadata directly within the document. In this case, the XML-based standard Resource Description Format (RDF) is your best bet. RDF is not tied to a particular metadata standard, so you can apply it to a variety of requirements.

Metadata can also enable access control. For example, you could use explicit attributes of a text resource - such as a copyright or license agreement - to manage the distribution of content. For example, corporate libraries that license electronic subscriptions to business and scientific journals might need to track the number of article downloads or restrict access to particular departments or a fixed number of concurrent users. Unlike content metadata, access-control metadata exists at several logical points, such as at the document, journal, magazine, or publisher level.

Quality control and document ranking is another (and often overlooked) use for metadata. Not all texts are created equal, and your information retrieval programs should reflect that fact. For example, you can safely assume an article from The Wall Street Journal on foreign investments in Latin America is accurate, but what about a piece from an obscure Web site dedicated to the same topic?

Even internal documents vary in importance. A final report to the COO should have more weight than draft memos circulating among analysts, yet the memos could easily rank higher in a search based simply on word frequencies.

Metadata is not limited to content description, access control, and quality control; you can also extend it to include automatically generated summaries and clustering data. Depending on the application, however, both summary and clustering information might be generated more effectively on an as-needed basis rather than explicitly stored.

PRINCIPLE 2: KNOW THE USER

Representing a user's long-term interests through a profile is yet another key to improving the precision and recall of information retrieval. Profiles, which are explicit representations of these interests, generally use the same representation schemes (such as keyword vectors) as the metadata describing the contents of documents.

Tsvi Kuflik and Peretz Shoval, who research information filtering techniques at Ben-Gurion University in Israel, have identified six different kinds of profiles:

• User-created profiles are the easiest to implement but put the burden on the user to create and maintain them.
• System-generated profiles analyze word frequencies in relevant documents to identify patterns indicative of interesting texts.
• System- plus user-created profiles start with an automatically generated profile that the user can subsequently adjust.
• Neural-net profiles are trained using interesting texts provided by the user to output relevancy rankings for other texts.
• Stereotype models of interests shared by a large group of users can provide the basis for building individualized profiles.
• Rule-based filtering implements explicit if-then rules to categorize content.

Each of these techniques has its benefits and drawbacks, such as requiring manual updating by users when interests change or slow adaptation to such changes. But whichever one you use, the generated profile provides a long-term resource for filtering, disambiguation, and document gathering.

PRINCIPLE 3: CONTROL ACCESS TO CONTENT

In contrast to the free-flowing information model of the Web, effective portals require controlled access to content because users are more likely to share information when they know it is distributed within the bounds of well-defined security business rules. In general, content can be grouped into three broad access control areas: open-access information, license-restricted information, and privileged information.

• Open-access information is freely available to all portal users. News feeds, press releases, product catalogs, and other publicly available information fall into this category.
• Access to license-restricted information is defined by agreements with content providers and applied to significant portions of all available content, such as the entire digital library of a professional association. In these cases, you can adequately control access through basic user authentication or IP address verification.
• Privileged information is the most challenging because access is granted on a need-to-know basis. For example, attorneys working on negotiations with one client will need access to related documents, while others working in the same office working on the same type of negotiations but with another firm should not be privy to those same texts. In such situations, the advantages of database-oriented applications become apparent. Relational databases natively provide role and privilege models for access control, and you can implement programmatic procedures for finer grained access control based on content metadata.

When content is adequately protected, you can then turn your attention to creating a navigable repository.