Text mining

Elizabeth D Liddy is with the Center for Natural Language Processing in the academy of Information, Studies at Syracuse University.

Text mining is the proces of analyzing naturally occurring true copy for the purpose of discovering and capturing semantic information for insertion and storage in what I'll call a Knowledge Organization manner of making (KOS) with the ultimate goal of enabling knowledge discovery via either textual or visual access for use in a wide range of significant applications.

Text mining is appropriately-considered a subspecialty of the broader domain of Knowledge Discovery from Data (KDD) which in revolve can be defined as the computational proces of extracting useful information from massive amounts of digital data through mapping low-level data into richer, more abstract forms and by the agency of detecting meaningful patterns implicitly not away in the data. KDD, which is typically administrationed on structured, relational databases, has data mining as single of its sub-tasks. While data mining has become the more popular limit it is in fact alone one of the steps within the KDD proces The filled KDD process includes data storage and access, data cleansing, pattern detection and extraction, and data interpretation, while data mining leaves more narrowly to the particular pace of applying specific algorithms for detecting and extracting patterns.

Text mining has reach forthed the applicability of KDD dramatically by dint of the use of sophisticated natural language processing [www.asis.org/Bulletin/ Apr-98/liddy.html]. This means that there is no ne to limit KDD to alone that information that is available in structur databases; nor do the knowledge bases of interest ne to be manually raiseed Given that much of the information of value for mining already resides in naturally occurring body s (or can be elicited as text) NLP provides the necessary techniques for true copy mining to extract knowledge automatically from these body s

Organizations interested in accomplishing knowledge management have begun to realize that a substantial proportion of knowledge that urgencys to be exploited and utilized already exists in textual form. A small in number examples of the range of information that is typically available within an organization include e-mail from customers, intranet memo and briefings, internal technical reports and patents, as well as newspaper and freshs wire stories about competitors as well as external views of the organization. Therefore, true copy mining, the process of analyzing body s to extract information useful for the one and the other discovery of patterns and stretchs as well as confirmation of hypotheses, has begun to gain acceptance as a highly desirable technology.

As predicted by dint of the Gartner Group in 1998 text-- mining capabilities have begun to appear in leading information retrieval (IR) harvests Beginning with the release of IBM's Intelligent body Miner in 1998, the bar for IR has been raised, and novel IR products are now awaited to have at least a clustering capability that will collection texts according to similarity of satisfied if not providing full mining capabilities.

While more traditional definitions of IR focus upon document retrieval, a more expansive view of the goal of IR is that it should minimize the human resources required to find the necessary information to accomplish a goal by dint of

permitting users to bear their needs in the greatest in quantity convenient and expressive mode possible;

placing the load on the system to understand and adapt itself to users' needs; and

providing precise flows pre-analyzed by the system and determined to be precisely relevant.

If a user requires a simple answer, the ideal IR combination of parts to form a whole would provide just that - not a list of potentially relevant documents. If discovery of tends across a document collection is the goal, then an IR combination of parts to form a whole should be able to perform true copy mining.

This broader definition of IR recognizes that the information exigencys of many users, particularly in strategic intelligence, are of of the like kind range and sophistication that a with truth useful system must go beyond simple retrieval.

It must provide a broad range of information access and analytic capabilities. The way that true copy mining can accomplish this goal is from one side reliance on NLP. NLP consists of a range of computational techniques for analyzing and representing naturally occurring body s at all levels of linguistic analysis to achieve human-like language processing that can support this kind of analysis.

A entirely featured Information Access & Analytic combination of parts to form a whole is one that combines one as well as the other IR and text mining capabilities. of that kind a technology would

detect the specific sources that contain information worth mining;

recognize and extract meaningful entities that transmit valuable knowledge;

produce a semantic interpretation of the information;

store the semantically interpreted information in an efficient data structure; and

provide means for easy access and utilization of this knowledge base for fresh insights or for utilization in decisionmaking tasks.

Potential uses of of the like kind a text mining capability include those essential to strategic or competitive intelligence. For example, true copy mining would enable a company to proces public novels sources, extracting information about their competitors' replications to events. This capability would then enable a strategic analyst within the company to erect a model of how each competitor reacts to specific stimuli and thereby enable the company to predict in what way each competitor would react in a similar novel situation. As a next pace it would enable a strategic analyst to characterize what a "generic" competitor would gaze like. Based on daily tracking of the company's known competitors, the analyst could build a type of the characteristics, actions and circumstances that would define a type of a generic competitor for that company. The design could be used as a daily profiler and extractor to mine newsfeed and recognize novel competitors.