JPH02158871A - Document sorting device - Google Patents

Abstract

PURPOSE:To sort documents based upon the number of concepts without generating dispersion by providing the device with a keyword information volume storing means, a concept feature extracting means and an inter-document distance calculating means. CONSTITUTION:The keyword information volume storing means 1 finds out the self-information volume of each keyword by prescribed calculation based upon the keyword appearance frequency of a document data base or the like, the concept feature extracting means 2 finds out the number of concept features in each document from the self-information volume and the inter-document distance calculating means 3 sorts documents in accordance with a different between plural concept feature values. Thus, the documents are automatically sorted through the calculation processing of respective means in accordance with the frequency of respective keywords. Consequently, ordinary manual operation can be omitted and document sorting having no dispersion and based upon the number of concept can be constructed.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to a document classification device, and more particularly, to a document classification device that obtains conceptual features of a document based on keywords included in the document and classifies documents based on the conceptual features. .

[Prior Art] In order to automatically classify documents into predetermined fields, a method is known in which a chi-square value is used to check the bias of keywords and classify documents. Such a classification method is described in Tamura et al., "Automatic classification of documents using statistical methods" (Proceedings of the 36th National Conference on Information Processing, 1987).
, O and Chikio Hayashi, “Methods of Quantification” (Toyo Keizai Shimbun,
1974).

The chi-square test is a method for classifying documents by determining the chi-square value as an index indicating the bias in the appearance frequency of keywords depending on the field. The chi-square value is calculated by using the frequency value of each keyword and the total number of keywords in each field or the frequency value of the keyword, assuming that it is a unique phenomenon, as the theoretical frequency, and calculating the difference between the actual value and the normal value. It has become.

The above-mentioned document (■) describes a method for automatically classifying documents into predetermined categories using a chi-square test. Since this method uses the bias in the appearance frequency of keywords, it is necessary to prepare data for classification by classifying a large amount of sample data in advance by field and calculating chi-square values.

Reference (2) is also a statistical method that uses chi-square values, and is a method for looking at correlations between multiple fields.

[Problems to be Solved by the Invention] The method described in the above-mentioned document ■■ still requires manual work to classify sample data. therefore,
There is a problem of interference due to variations and inappropriateness of manual classification.

In addition, the latter has the problem that it is difficult to determine the axis for classification.

In order to solve the above problems, the present invention calculates the conceptual characteristics of each document from the frequency values of keywords included in the document.
It is an object of the present invention to provide a document classification device that classifies documents accordingly.

[Means for Solving the Problems] In order to achieve the above object, the present invention provides a keyword information storage that holds the self-information amount of each keyword calculated using the appearance frequency value of the keyword in the document database. a conceptual feature extraction means for calculating the conceptuality trace amount of each document using the self-information amount of the keyword; and an inter-document distance calculation means for calculating the distance between documents according to the difference in the conceptuality trace amount between documents. have

The inter-document distance calculation means classifies documents based on the distance between documents.

[Function] According to the present invention, a predetermined calculation is performed to obtain the self-information amount of each keyword based on the keyword appearance frequency in the keyword information storage means or a document database, and the concept feature extraction means extracts a predetermined amount of self-information from the self-information amount. The conceptuality trace amount of each document is determined by the calculation of Since documents are classified, conventional manual work is no longer required, and document classification based on conceptual quantities without variation can be constructed.

[Example] Embodiments of the present invention will be specifically described with reference to the drawings.

An embodiment of a document classification device according to the invention is shown in the figure.

The keyword information storage unit 1 extracts keywords from the input unregistered sentence 6Q, calculates the probability of appearance of the keyword from its appearance frequency as described later, and stores the logarithm value as the keyword information amount I.

The conceptual feature extraction unit 2 inputs the keyword information amount from the keyword information storage unit 1, and outputs the sum as the conceptual trace amount C(q) of the document Q. The inter-document distance calculation unit 3 inputs and stores the conceptual trace amount C(q) of each document from the conceptual feature extraction unit 2, calculates the conceptual distance between two documents, and clusters (classifies) documents with close conceptual distances. ) and store various classifications in the document database 4. Each functional unit is connected by a data bus a-C that transfers data generated by each unit.

In general, for keywords registered in a keyword collection such as a thesaurus, the appearance frequency of each keyword can be defined by the number of documents in which the keyword appears, the total number of appearances in all documents, and the like. Now, if P, which is the appearance frequency of keyword KEY i normalized by the total number of keywords, is taken as the appearance probability of keyword KEY i, then the system that associates the appearance probability P with keywords is a complete event system and can be expressed as follows.

However, ΣP=1.

1;1 Here, the self-information amount I (KEYi) of KEYi can be expressed by the following equation.

1 (KEYi) = -log Pi
=. (1) Since the amount of self-information maintains additivity, the amount of combined information possessed by KEYi and KEYj is expressed by the following equation.

1 (KEYi, KEYj) = I (KEYi)
+ l (KEYj) = -log Pi -lo
g Pj - (2) The keyword information storage unit 1 stores the unregistered document Q in the document database 4 from the conceptual feature extraction unit 2.
Each keyword of one document Q is inputted from the data bus a via the data bus a, and EYi is determined as its appearance probability, and the self-information amount 1 (EYi) of the keyword is calculated and held using equation (1). When a thesaurus is prepared, the probability of appearance of a keyword is determined for each keyword classification item in the thesaurus, and the amount of self-information can be calculated using equation (1).

Let q be a keyword set of a certain document Q, and express its conceptual feature as (:(q)), then it is given by.

Furthermore, in a thesaurus with existing classification items, conceptual features can be treated as vectors. As the simplest example, consider an M-dimensional vector Cv in a thesaurus having M classification items.

Now, if the set of keywords belonging to the R-th classification item is r, then the R element CVr(q) of the conceptual feature vector CV(q) of document Q is: This means calculating the sum total of Pi when both are included and included in the R-th classification item.

The self-information amount ■ of each keyword of the document Q is inputted from the keyword information amount storage unit l, and the conceptual feature extraction unit 2 extracts the conceptual feature amount C(q) or C using equation (3) or equation (4).
Vr(q) is calculated and output to the inter-document distance calculation unit 3 via data bus b.

The amount of conceptual information obtained by equation (3) is the sum of the amount of keyword information that a certain document has, and only indicates the amount of self-information added to that document. In this case, the amount of conceptual information indicates the degree of separation (ease of identification) of the document at the time of searching the document database, and it is also possible to classify documents based on the degree of separation.

However, it is usually possible to classify documents into existing classification categories depending on their contents. In such a case, (4)
Using the conceptual feature value vector of Eq. - In general, a database is considered to constitute an M-dimensional conceptual space with M classification items. Therefore, the concept of a document in such a database can be expressed as an M-dimensional vector consisting of M feature parameters. Furthermore, since the distance between any two conceptual feature vectors can be calculated, the degree of belonging of a certain document to a certain classification, the conceptual distance between two documents, etc. can be determined.

For example, if INC(k, Q) is the degree to which a document with a conceptual feature vector GV(q) belongs to classification K with keyword set k, then ING(k, q) = CVk(q)/ ΣCVr(
q) ... (5) Given by J.

Furthermore, if we assume that the conceptual distance between two documents with conceptual feature vectors CV(s) and CV(t) is (s, t), and calculate it using the city area distance, for example, D(s, t) = Σ l CVr (s) −(:Vr
(t) l ... (6) r is given by 1.

The inter-document distance calculation unit 3 calculates the conceptual feature amount C(q) or GV(
By inputting Q) and performing the calculation shown in equation (5), it is possible to determine the classification to which an unclassified document belongs.
) formula allows several classifications to be formed by groups of documents with close conceptual distances.The inter-document distance calculation unit 3 inputs the classification of the document Q into the document database 4. Since the classification generated at this time is a composite concept that combines the concepts of several existing classification items, a new classification system that is oriented to the document concept itself and is not bound by existing classification items is naturally constructed.

A method for classifying similar documents using equation (6) will be specifically explained.

As mentioned above, when performing document classification on existing classification items, it is sufficient to calculate the degree of belonging to each classification K (INC(k, q)) using equation (5). When used, it becomes possible to construct a new classification system using existing classification items.

First, calculate the conceptual distance between each document for all documents to be classified.
A document (document S) is selected, and documents whose conceptual distance to the document is smaller than a predetermined threshold, that is, documents that are conceptually similar to the document are extracted. Expressing the set of extracted documents T using a formula, if one document S and the keyword sets included in T are t and s, respectively, (TID (s, t) < θ) (where D (s, 5) =0 is clear from (6), and the document S is always included in the set T.) If this operation is performed for all documents, a set of similar documents equal to the number of documents will be created. These similar document sets are arranged in descending order according to the number of elements (number of documents) in the set, and similar document sets with the required number of classifications are selected in descending order of the number of documents. This selection may be limited by the number of classifications or the number of documents.

The maximum number that can be classified is the number of documents. In this case, the number of documents included in each classification is l, but there may be cases where such classification is optimal.

[Effects of the Invention] According to the present invention, conceptual features can be calculated using keyword extraction or classification of existing keyword collections, so there is no need to create evaluation data before classifying unregistered documents.

Since a classification is constructed from a group of documents with close conceptual distance, it has the excellent effect of naturally constructing a new classification system that is oriented to the document concept itself and is not limited by existing classification items.

[Brief explanation of the drawing]

The figure is a functional block diagram showing an embodiment of the document classification device of the present invention. Explanation of symbols of important parts 1: Keyword information storage unit, 2: Conceptual feature extraction unit, 3: Inter-document distance calculation unit, 4: Document database.

Claims (1)

[Scope of Claims] 1. Keyword information storage means for storing the self-information amount of each keyword calculated using the appearance frequency value of the keyword in the document database; conceptual feature extraction means for calculating the conceptual feature amount of the document; and inter-document distance calculation means for calculating the distance between documents according to the difference in the conceptual feature amount between the documents; A document classification device characterized by classifying documents based on the distance between them. 2. Keyword information storage means for storing the self-information amount of keywords calculated using the appearance frequency value of keywords for each keyword classification item of the thesaurus used in the document database, and the keyword information for each keyword classification item. a conceptual feature extraction means for obtaining a vectorized sum of quantities as a conceptual feature; and an inter-document distance calculation means for obtaining a distance between documents according to a difference in the conceptual feature between documents; A document classification device characterized in that the distance calculation classifies documents based on the distance between the documents.