JP4630911B2 - Document classification apparatus, document classification method, and computer-readable recording medium storing a program for causing a computer to execute the methods - Google Patents

Description

  The present invention relates to a document processing apparatus that outputs a plurality of input document data for display or printing in a predetermined format, a document processing method, and a computer-readable recording medium that records a program that causes a computer to execute the method. . Further, the present invention classifies a plurality of input documents based on the contents of the documents, and particularly refines a classification category (system) calculated at the time of document classification, a document classification method, and a document classification method The present invention relates to a computer-readable recording medium on which a program for causing a computer to execute the method is recorded.

  In recent years, various document classification devices and document search devices have been developed. In addition, the spread of network technology such as the Internet has made it possible to access a large amount of electronic documents in Japan and overseas, and the amount of information stored electronically on business has increased dramatically in proportion to this. There is an increasing need for intelligent work such as classifying a large amount of collected document information into meaningful categories (systems).

  The purpose of the semantic classification of these large amounts of document information is as follows. First, improvement in searchability can be considered. This is because a huge group of documents can be searched by using a classification name (content name) as a clue, so that the search becomes relatively easy.

  Second, it is possible to grasp the entire information group. This grasps what kind of contents (individual classification) the entire document group is composed of. However, when an operator manually classifies a large amount of document information, it can be accurately classified, but the human and time costs associated with classification become enormous, so the amount of documents accumulated in recent years Due to the enormous volume, document information automatic classification devices have been proposed.

  As a prior art of an automatic document classification device, for example, as described in Japanese Patent Laid-Open No. 7-36897, a document is regarded as a document vector characterized by a word, and these document vectors are converted using a clustering method. Some groups perform automatic classification of documents based on grouped document vectors.

  In the “Projects for Effective Document Clustering” (author name: Hinrich Schutze and Craving Silverstein, conference name: ACM, paper name: Proceedings of SIGIR, page: 74-81, publication year: 1997) Some document classifications are implemented. As other methods, a method using a probabilistic approach can be considered.

  In recent years, with the spread of the Internet and the like, it has become possible to access a large number of document groups, and as a result, there is a need to be able to use the document groups efficiently and based on the intentions of various users. Is growing. For this reason, intelligent work has started to be performed, in which a large number of document groups are classified into meaningful categories and the structure of the document groups is grasped. However, when such classification work is performed manually, the human and time costs are enormous, and only the classifier has knowledge for classification. As a result, the classification criteria will change.

  Therefore, there is a demand for a document classification device that can automatically classify a document group according to a classification standard for human classification, and the document classification device is described in, for example, Japanese Patent Laid-Open No. 7-114572. In addition, there is a technique that enables automatic classification using semantic differences by automatically extracting feature vectors of words from a document and classifying the documents based on the feature vectors.

  However, since the above-described prior art document classification device is a method for statistically classifying a document placed in a multidimensional space composed of words, the classification result is statistical from the viewpoint of the so-called behavior of words. In other words, each cluster (a subset of each classified document) calculated as a result of classification may not be understood by the operator (user).

  In addition, it is difficult to define the optimum classification result because the classification result is optimal depending on the characteristics of the document set to be classified and the purpose of the user's work. . In particular, with respect to grasping the entire information group, there is a problem in that it is difficult to obtain the result desired by the operator with a single classification operation because the classification required by various operators' intentions is different. .

  As described above, the document classification result can be interpreted as including many so-called noises, and there is a problem that only a part of the result is often beneficial to the operator.

  In addition, since these conventional techniques do not consider the unit of the document, if a document has a structure separated by one or more paragraph marks or titles, a plurality of documents are included in one document. As a result, it becomes difficult for the user to understand the classification category, or the category is limited to a specific topic or a specific meaning. There is a problem of being classified into a category different from the category.

  Note that the context-dependent automatic classification apparatus disclosed in Japanese Patent Laid-Open No. 6-176064 discloses an apparatus that attempts to increase classification accuracy by performing automatic document classification in consideration of document paragraph information. However, it does not solve the above problem.

  In addition, document processing apparatuses such as the above-described prior art document classification apparatus and document search apparatus simply have a function of classifying a document or searching for a document. No consideration has been given to the analysis of the underlying hidden information, and there has been a problem that the analysis of the hidden information inherent in the document group must be performed using another analysis device.

  Also, the reason why the operator who performs information analysis performs the classification work and the search work is that the result is not the purpose in these work, but merely the progress of the information analysis work. Usually, in order to make the results easier to understand, the information contained in the original document is utilized to the maximum, and the results are rearranged, aggregated / statistically processed, etc. It is necessary to repeatedly perform various processes such as tabulating and graphing to derive meaningful information analysis results.

  In addition, spreadsheet software may be used in the analysis of information for numeric data, but spreadsheet software was originally developed with the intention of handling numeric data, In the analysis work related to the character data, especially the meaning of the document, it was not possible to exert a sufficient effect.

  In order to solve the above-described problems caused by the conventional example, the present invention does not simply perform classification work or search work as a fixed function in the analysis work related to the meaning of the document, and output the result. It is a first object of the present invention to provide a document processing apparatus, a document processing method, and a computer-readable recording medium on which a program for causing a computer to execute the method is provided that can provide support for the entire information analysis work.

  Further, the present invention provides a document classification apparatus, a document classification method, and a method thereof capable of gradually collecting what kind of contents are included in an arbitrary document set in order to solve the problems caused by the above-described conventional example. A second object is to provide a computer-readable recording medium recording a program to be executed by a computer.

  In addition, in order to solve the problems of the conventional example described above, when a plurality of topics and meanings are included in one document, the present invention is classified into a category limited to a specific topic or meaning. Document classification device, document classification method and program for causing a computer to execute the method can be recorded so that the user or the classification category can be well understood by being not classified into a category different from the category intended by the user. A third object is to provide a computer-readable recording medium.

In order to solve the above-described problems and achieve the object, the document classification apparatus according to the first aspect of the present invention is an input means for inputting document data in a document classification apparatus that classifies documents based on the contents of the document. And language analysis means for analyzing the document data input by the input means to obtain language analysis information, based on the language analysis information obtained by the language analysis means , a word as a feature dimension, and the feature dimension as Document generation based on similarity between a vector generation unit that generates a document feature vector, which is column vector information representing the document data, and a document feature vector generated by the vector generation unit in a document expression space as a row Classifying means for generating a document subset and a cluster feature for calculating a cluster feature that is a feature of the document subset generated by the classification means. A star feature calculation means, a cluster selection instruction means for selecting a desired subset from the document subsets generated by the classification means, and a document feature vector for storing the document feature vectors generated by the vector generation means Storage means and vector correction means for correcting the document feature vector stored by the document feature vector storage means so that the document feature vector of the document belonging to the subset selected by the cluster selection instruction means is removed. And the centroid of the subset calculated from the subset selected by the cluster selection instruction means from the document expression space used when judging the similarity between the document feature vectors stored in the document feature vector storage means a feature quantity that represents a row vector of feature dimensions and selection by the cluster selection instructing means A row vector of feature dimensions based on document feature vector corresponding to the document included in the subset that is, of the feature dimensions of a high degree of similarity the document representation space determined, a row vector corresponding to the characteristic dimension and the determination Document expression space correction means for correcting by deleting from the feature dimension of the document expression space, and the classification means uses the document expression space corrected by the document expression space correction means, and the vector correction means. The document is classified based on the similarity between the document feature vectors corrected by the above.

According to the first aspect of the present invention, it is possible to eliminate the influence of a known cluster, and to eliminate the cluster formation feature selected by the operator as a result of the previous classification execution at the next classification execution. And a new cluster can be generated in the excluded state.

The document classification device according to a second aspect of the present invention is the document classification device according to the first aspect, further comprising display means for displaying the cluster feature calculated by the cluster feature calculation means, Selection information giving means for giving selection information indicating selection when all or some of the documents belonging to the subset of documents generated by the classification means are selected, the display means, The cluster feature is displayed, and the selection information given by the selection information giving means is displayed.

According to the second aspect of the present invention, it is possible to improve the identifiability of documents used in multiple and the identities of documents that have never been selected.

According to a third aspect of the present invention, there is provided the document classification method according to the third aspect of the present invention, in which the document classification method performs document classification based on the content of the document. A language analysis step for analyzing input document data to obtain language analysis information, and a document expression space having words as feature dimensions and rows as the feature dimensions based on the language analysis information obtained by the language analysis means In the above, a vector generation step of generating a document feature vector which is column vector information representing the document data, and a document is classified based on the similarity between the document feature vectors generated by the vector generation step, A classification step for generating a subset, and a cluster feature calculation step for calculating a cluster feature that is a feature of the subset of the document generated by the classification step; A cluster selection instruction step for selecting a desired subset from a subset of documents generated by the classification step, a document feature vector storage step for storing a document feature vector generated by the vector generation step, A vector correction step of correcting the document feature vector stored in the document feature vector storage step so that the document feature vector of the document belonging to the subset selected by the cluster selection instruction step is removed; and the document feature The centroid of the subset calculated from the subset selected by the cluster selection instruction step from the document expression space used when judging the similarity between the document feature vectors stored in the vector storage step is the row of the feature dimension. a feature amount representing a vector, containing the subset selected by the cluster selection instruction step Determine the row vector of feature dimensions based on document feature vectors corresponding to documents, of feature dimensions of a high degree of similarity the document representation space, a row vector corresponding to the characteristic dimension and the determination wherein the document representation space A document expression space correction step for correcting by deleting from a dimension, and the classification step uses the document expression space corrected by the document expression space correction step, and the document feature corrected by the vector correction step. Documents are classified based on the similarity between vectors.

According to the third aspect of the present invention, the formation characteristics of the cluster selected by the operator as a result of the previous classification execution can be excluded at the next classification execution, and a new cluster can be generated in the excluded state. Can do.

The document classification method according to a fourth aspect of the present invention is the document classification method according to the third aspect of the invention, further comprising a display step of displaying the cluster feature calculated by the cluster feature calculation step, A selection information giving step for giving selection information indicating that the document is selected when all or a part of the documents belonging to the subset of documents generated by the classification step is selected; The cluster feature is displayed, and the selection information given by the selection information giving step is displayed.

According to the fourth aspect of the present invention, it is possible to eliminate the influence of a known cluster, and to eliminate the formation characteristics of the cluster selected by the operator as a result of the previous classification execution at the next classification execution. And a new cluster can be generated in the excluded state.

  A document classification method according to a thirty-sixth aspect of the invention is the document classification method according to the thirty-first to thirty-fifth aspects of the invention, wherein the classification system generation step includes a cluster feature selected by the selection instruction step, and a subset of the document A component of a classification system is generated based on all or part of a document group belonging to a subset of documents selected from the above and / or information created by an operator.

  According to the thirty-sixth aspect of the present invention, the contents of the cluster can be easily grasped, and the operator's own classification system can be easily generated, so that the utility value of the classification system can be improved.

  A document classification method according to a thirty-seventh aspect of the present invention is a document classification method for classifying a document group according to the contents of the document. The document is divided based on a reference, one or a plurality of divided document data is generated from one document data, a document-divided document correspondence map indicating a correspondence between the document data and the divided document data is generated, The divided document data is classified, divided document classification result information is generated, and the document data classification result information is generated using the document-divided document correspondence map and the divided document classification result information.

  According to the present invention of claim 37, when a plurality of topics and meanings are included in one document, it is classified into a category limited to a specific topic or meaning or intended by the user. It is not classified into a category different from the category, so that the user can understand the classification category well. Further, since the position of the divided document in the pre-division document (affiliation document) is also shown, the user can efficiently read the portion to be read in the document group.

  The recording medium according to the invention of claim 38 records the program for causing a computer to execute the method described in claims 24 to 37, so that the program can be read by a machine. The operation of 37 can be realized by a computer.

  As described above, according to the invention of claim 1, the document storage means for storing the input document data, the selection means for selecting all or part of the document data stored by the document storage means, Feature extraction means for extracting data relating to character string features from all or part of the document data selected by the selection means; and data on the document data based on data relating to character string features extracted by the feature extraction means. In the analysis work related to the meaning of the document, since the processing means for processing all or part of the data and the output means for outputting all or part of the document data processed by the processing means, The effect of obtaining a document processing apparatus that can support not only the results but also the overall information analysis work Unlikely to.

  According to the invention of claim 2, the output means sets item value setting means for setting a plurality of item values based on all or part of the contents of the document data processed by the processing means; A totaling unit that totalizes all or part of the document data for each item value set by the item value setting unit, and the whole or a part of the document data has at least one axis as the item value Since it is expanded and output in a table format, the processing results can be expressed as a cross table with a simple operation, and the contents of information can be easily grasped, so analysis work related to the meaning of the document In this case, it is possible to obtain a document processing apparatus capable of providing support for the entire information analysis work rather than simply outputting the result.

  According to a third aspect of the present invention, the output unit further includes a part of the document data before being processed by the processing unit, all or part of the document data processed by the processing unit. Since all or part of the data is output, the target data to be processed and other data are displayed at the same time. By checking this, the target range for processing can be determined accurately and easily. In the analysis work related to the meaning of the above, it is possible to obtain a document processing apparatus capable of providing support for the entire information analysis work rather than simply outputting the result.

  According to the invention of claim 4, since the document storage means further stores all or part of the document data processed by the processing means, it is handled in the same manner as other data thereafter. Therefore, in the analysis work related to the meaning of the document, it is possible to obtain a document processing apparatus capable of providing support for the entire information analysis work, not simply outputting the result.

  According to the invention of claim 5, the selecting means further selects all or part of the document data output by the output means, so that all or part of the document data output by the output means is selected. Can be subject to further analysis, and a variety of advanced information analysis work can be performed. Therefore, in the analysis work related to the meaning of the document, not only the results are output, but the support for the entire information analysis work It is possible to obtain a document processing apparatus capable of performing the above.

  According to a sixth aspect of the present invention, the document storage means further stores data relating to the contents of the processing, thereby preventing loss of data relating to the contents of the processing and facilitating management of the data. In addition, since it is possible to grasp the settings used for processing and the processing results by associating them, in the analysis work related to the meaning of the document, not only outputting the results but also analyzing the information There is an effect that a document processing apparatus capable of providing support for the entire work can be obtained.

  According to the invention of claim 7, the input means inputs the document data, the language analysis means analyzes the document data input by the input means to obtain language analysis information, and the vector generation means includes: A document feature vector for the document data is generated based on language analysis information obtained by the language analysis unit, and a classification unit classifies the document based on a similarity between the document feature vectors generated by the vector generation unit. Then, a subset of the document is generated, the cluster feature calculation means calculates a cluster feature that is a feature of the document subset generated by the classification means, and a classification system storage means calculates by the cluster feature calculation means The cluster feature is stored as a component of the classification system, so the cluster can be obtained and the similarity between the cluster centroids can be used. Cluster structuring and systematization can be performed based on the contents of the cluster, thereby obtaining a document classification device that can gradually collect what contents are included in an arbitrary document set. There is an effect.

  According to the invention of claim 8, the input means inputs the document data, the language analysis means analyzes the document data input by the input means to obtain language analysis information, and the vector generation means includes: A document feature vector for the document data is generated based on language analysis information obtained by the language analysis unit, and a classification unit classifies the document based on a similarity between the document feature vectors generated by the vector generation unit. Then, a subset of the document is generated, the cluster feature calculation means calculates a cluster feature that is a feature of the document subset generated by the classification means, and the display means is calculated by the cluster feature calculation means. The cluster feature is displayed, and the cluster selection instruction means selects a desired subset from the document subset generated by the classification means, and the classification system Since the memory means stores the subset of documents selected by the cluster selection instruction means as a component of the classification system, the cluster structure closer to the operator's intention is made using only the selected cluster. Systematization can be performed, thereby producing an effect of obtaining a document classification device that can gradually collect what content is included in an arbitrary document set.

  According to the invention of claim 9, in the invention of claim 8, the document feature vector storage means stores the document feature vector generated by the vector generation means, and the vector correction means stores the document feature vector storage. The document feature vector stored by the means is corrected so as to be the result of removing the document feature vector of the document belonging to the subset selected by the cluster selection instruction means, and the classification means is corrected by the vector correction means. Since the document is classified based on the document feature vector, it is possible to generate a new cluster that eliminates the influence of the known cluster, and what content is included in an arbitrary document set It is possible to obtain a document classification device that can gradually collect.

  According to the invention of claim 10, in the invention of claim 8, the document feature vector storage means stores the document feature vector generated by the vector generation means, and the document expression space correction means has the document feature. Correcting the document expression space when judging the similarity between the document feature vectors stored by the vector storage means based on the feature amount calculated from the subset selected by the cluster selection instruction means, and the classification means, In order to classify the document based on the similarity between the document feature vectors generated by the vector generation unit using the document expression space corrected by the document expression space correction unit, the result of the previous classification execution, the operator The formation characteristics of the selected cluster can be excluded at the next classification execution, and a new cluster can be generated in the excluded state. Thereby, an effect that what the progressively document classification apparatus capable of collecting content includes any document set is obtained.

  According to the invention of claim 11, in the invention of claim 9, the document feature vector storage unit stores the document feature vector generated by the vector generation unit, and the document expression space correction unit stores the document feature. A document expression space used when judging the similarity between document feature vectors stored by the vector storage means is corrected based on the cluster feature selected by the cluster selection instruction means, and the classification means corrects the document expression space. In order to classify the document based on the similarity between the document feature vectors corrected by the vector correction means using the document expression space corrected by the means, the influence of the known cluster is eliminated, and the previous time As a result of the classification execution, the cluster formation feature selected by the operator can be excluded at the next classification execution, It can generate clusters, thereby, an effect that what the progressively document classification apparatus capable of collecting content includes any document set is obtained.

  According to the twelfth aspect of the invention, in the invention of the eighth or tenth aspect, the selection information adding means selects all or a part of the documents belonging to the subset of the documents generated by the classification means. Identification information indicating that the document has been selected is displayed, and the display unit displays the cluster feature and also displays the selection information provided by the selection information addition unit, so that identification of a document to be used multiple times is performed. And a document classification device capable of gradually collecting what kind of contents are included in an arbitrary document set. There is an effect.

  According to a thirteenth aspect of the present invention, in the eighth to twelfth aspects, the classification system storage means includes all or a part of documents belonging to a subset of documents selected by the selection instruction means, Since cluster features and / or arbitrary information created by the operator is stored as a component of the classification system, it is easy to understand the contents of the cluster, and the operator's own classification system can be easily generated. It is possible to improve the utility value, and it is possible to obtain a document classification device that can gradually collect what content is included in an arbitrary document set.

  According to the fourteenth aspect of the present invention, in a document classification apparatus for classifying a document group according to the contents of the document, a document input unit for inputting the document data group, and a predetermined value for each document in the input document data group A document-divided document correspondence map indicating the correspondence between the document data and the divided document data, and a document dividing unit that divides the document based on the criteria of the document and generates one or a plurality of divided document data from one document data Generating a document-divided document correspondence map, a divided document classification unit for classifying the divided document data, and a divided document classification result generation for generating divided document classification result information based on a classification result by the divided document classification unit Classification result information for generating classification result information of the document data using the means, the document-divided document correspondence map, and the divided document classification result information If there are multiple topics or meanings in one document, it is classified into a category limited to a specific topic or meaning, or a category intended by the user. Are not classified into different categories. Therefore, the user can understand the classification category well, and the position of the divided document in the pre-division document (affiliated document) is also shown. Has an effect that a document classification device capable of efficiently reading a portion to be read in a document group can be obtained.

  According to a fifteenth aspect of the present invention, in the fourteenth aspect of the present invention, the document storage unit that stores the document data, the divided document storage unit that stores the divided document data, and the document-divided document correspondence map generation Document-divided document correspondence map storage means for storing the document-divided document correspondence map generated by the means, and the same document data without regenerating the divided document data and the document-divided document correspondence map In addition, it is possible to efficiently obtain classification results with different parameters such as the number of classifications, classification method, and various settings at the time of classification, and data necessary for classifying document data and generating classification results Is saved, so that the user can have time freedom for the classification work and reanalyze the document classification performed in the past at any time. Ukoto can also document classification apparatus an effect that is obtained.

  According to a sixteenth aspect of the present invention, in the fifteenth aspect of the invention, since the divided document classification result storing means for storing the divided document classification result information generated by the divided document classification result generating means is provided, In addition to the effects of the invention, once classification is performed, the classification result can be expressed in various formats such as text expression, table expression, graph expression, etc., and divided document classification result information is saved This allows users to have time flexibility in performing classification and analyzing classification results, and reanalyzing document classification results performed in the past in various representation formats at any time. The document classification device that can also be obtained is obtained.

  According to the invention of claim 17, in the inventions of claims 14 to 16, since the plurality of divided document data generated by the document dividing means includes the document data itself before the division, the user can In addition to the detailed document data classification structure obtained by classifying the document data that has been classified, the classification structure that combines the rough macro classification structure obtained as a result of classification of the original document data itself is obtained. Thus, there is an effect that a document classification device that can be used is obtained.

  According to the invention of claim 18, in the inventions of claims 14 to 17, the document dividing unit is configured to divide the document data based on the structure information of the document data. Therefore, it is possible to obtain a document classification apparatus that can appropriately perform document classification and that can appropriately perform document classification in which the detailed classification structure of document data is known.

  According to a nineteenth aspect of the invention, in the fourteenth to seventeenth aspects of the invention, the document element extracting means for extracting elements included in the document data, and the elements accompanying the elements extracted by the document element extracting means Element accompanying information extracting means for extracting accompanying information, and the document dividing means is an element extracted by the document element extracting means, or the element accompanying information extracted by the element and the element accompanying information extracting means, Since the document data is divided by using the document data, it is possible to obtain a document classification device capable of appropriately performing document classification that can understand the detailed classification structure of the document data.

  According to a twentieth aspect of the invention, in the inventions of the fourteenth to seventeenth aspects, the document dividing means divides the document data in accordance with the designated designated range. In addition, there is an effect that a document classification apparatus capable of performing document classification that can understand the detailed classification structure of document data is obtained.

  According to a twenty-first aspect of the invention, in any of the fourteenth to seventeenth aspects, the document dividing unit divides the document data based on the number of characters in the document data, the number of sentences, or both the number of characters and the number of sentences. Therefore, there is a high possibility that contents with different topics will be classified as different documents. Therefore, even in the present invention, a document classification device capable of performing document classification that can understand the detailed classification structure of document data is obtained. There is an effect.

  According to a twenty-second aspect of the present invention, in the fourteenth to twenty-first aspects, the document classification result generating means extracts information indicating document data and representative information associated with the document data as classification result information. Therefore, the user can obtain a document classification apparatus that can easily grasp the outline of the detailed classification structure of the document data and the overall structure.

  According to a twenty-third aspect of the invention, in the twenty-second aspect of the invention, the document classification result generating means uses information indicating divided document data and representative information accompanying the divided document data as classification result information. Since it is configured to be extracted and presented, the user can easily understand the details of the classification structure of the document data and the overall structure as well as which divided documents are attributed to the category. There exists an effect that an apparatus is obtained.

  According to a twenty-fourth aspect of the present invention, the document storing step for storing the input document data, the selecting step for selecting all or part of the document data stored by the document storing step, and the selecting step A feature extraction step for extracting data relating to character string features from all or part of selected document data, and all or part of the document data based on data relating to character string features extracted by the feature extraction step And an output step for outputting all or part of the document data processed by the processing step. In the analysis work related to the meaning of the document, the result is simply the result. The document processing method capable of providing support for the entire information analysis work can be obtained.

  According to the invention of claim 25, the output step sets an item value setting step for setting a plurality of item values based on the whole or part of the contents of the document data processed by the processing step; A totaling step of totaling all or part of the document data for each item value set by the item value setting step, and using all or part of the document data as item values as at least one axis Since the data is expanded and output in a table format, the processing results can be expressed as a cross table with a simple operation and the contents of the information can be easily grasped. In this case, it is possible to obtain a document processing method capable of providing support for the entire information analysis work rather than simply outputting the result.

  According to a twenty-sixth aspect of the present invention, the output step further includes the processing of the document data before being processed by the processing step, all or part of the document data processed by the processing step. Since all or part of the data is output, the target data to be processed and other data are displayed at the same time. By checking this, the target range for processing can be determined accurately and easily. In the analysis work related to the meaning of the above, it is possible to obtain a document processing method capable of providing support for the entire information analysis work, not simply outputting the result.

  According to a twenty-seventh aspect of the present invention, the document storing step further stores all or part of the document data processed by the processing step, so that it is handled in the same manner as other data thereafter. Therefore, in the analysis work related to the meaning of the document, it is possible to obtain a document processing method capable of providing support for the entire information analysis work, not simply outputting the result.

  According to the invention of claim 28, since the selecting step further selects all or part of the document data output by the output step, all or part of the document data output by the output step. Can be subject to further analysis, and a variety of advanced information analysis work can be performed. Therefore, in the analysis work related to the meaning of the document, not only the results are output, but the support for the entire information analysis work It is possible to obtain a document processing method capable of performing.

  According to the invention of claim 29, since the document storing step further stores data relating to the contents of the processing, the loss of data relating to the contents of the processing is prevented and management of the data is facilitated. In addition, since it is possible to grasp the settings used for processing and the processing results by associating them, in the analysis work related to the meaning of the document, not only outputting the results but also analyzing the information There is an effect that a document processing method capable of providing support for the entire work can be obtained.

  According to the invention of claim 30, the input step inputs document data, the language analysis step analyzes the document data input by the input step to obtain language analysis information, and the vector generation step includes: A document feature vector for the document data is generated based on the language analysis information obtained by the language analysis step, and the classification step classifies the document based on the similarity between the document feature vectors generated by the vector generation step. Then, a subset of the document is generated, the cluster feature calculation step calculates a cluster feature that is a feature of the document subset generated by the classification step, and the classification system generation step is calculated by the cluster feature calculation step. Since the components of the classification system are generated based on the cluster feature, the cluster can be obtained and the similarity between the cluster centroids can be obtained. And a document classification method that can gradually collect what content is included in an arbitrary document set. The effect is obtained.

  According to the invention of claim 31, the input step inputs document data, the language analysis step analyzes the document data input by the input step to obtain language analysis information, and the vector generation step includes: A document feature vector for the document data is generated based on the language analysis information obtained by the language analysis step, and the classification step classifies the document based on the similarity between the document feature vectors generated by the vector generation step. A subset of documents is generated, a cluster feature calculation step calculates a cluster feature that is a feature of the document subset generated by the classification step, and a display step is calculated by the cluster feature calculation step. The cluster feature is displayed, and the cluster selection instruction step selects a desired subset from the document subset generated by the classification step, and classifier Since the generation process generates the components of the classification system based on the cluster feature selected by the cluster selection instruction process, the cluster structuring closer to the operator's intention is made using only the selected cluster. Systematization can be performed, thereby producing an effect of obtaining a document classification method capable of gradually collecting what content is included in an arbitrary document set.

  According to a thirty-second aspect of the invention, in the thirty-first aspect of the invention, the vector correction step is corrected so that the document feature vector of the document belonging to the subset selected by the cluster selection instruction means is removed. Then, since the classification step classifies the document based on the document feature vector corrected by the vector correction step, it is possible to generate a new cluster excluding the influence of the known cluster, There is an effect that a document classification method capable of gradually collecting what content is included in an arbitrary document set can be obtained.

  According to a thirty-third aspect of the invention, in the thirty-first aspect of the invention, the document expression space correcting step determines the document expression space when judging the similarity between the document feature vectors generated by the vector generating step. Correction is made based on the feature amount calculated from the subset selected by the cluster selection instruction step, and the classification step is generated by the vector generation means step using the document expression space corrected by the document expression space correction step. Since the documents are classified based on the similarity between the document feature vectors, the cluster formation features selected by the operator as a result of the previous classification execution can be excluded at the next classification execution, and are excluded. A new cluster can be created with this, which allows you to gradually collect what content is included in any document set An effect that kind methods are obtained.

  According to a thirty-fourth aspect of the present invention, in the thirty-second aspect of the present invention, the document expression space modification step determines the similarity between the document feature vectors generated by the vector generation step. Correction based on the feature amount calculated from the subset selected by the cluster selection instruction step, and the classification step is corrected by the vector correction step using the document expression space corrected by the document expression space correction step. Since the documents are classified based on the similarity between the document feature vectors, the influence of the known clusters is eliminated, and the cluster formation features selected by the operator as a result of the previous classification execution are It can be excluded at the time of classification execution, and a new cluster can be generated in the excluded state, so that what content is included in an arbitrary document set Progressively document classification method capable of collecting or an effect that can be obtained.

  According to the invention of claim 35, in the invention of claim 31 or 33, all or part of the documents belonging to the subset of documents generated by the classification step is selected in the selection information adding step. Identification information indicating that the document has been selected is displayed, and the display step displays the cluster feature and also displays the selection information provided by the selection information addition step, so that multiple documents can be identified. And a document classification method that can gradually collect what content is included in an arbitrary document set. There is an effect.

  According to a thirty-sixth aspect of the invention, in the thirty-first to thirty-fifth aspects, the classification system generation step is selected from a subset of the document in addition to the cluster feature selected by the selection instruction step. Since the components of the classification system are generated based on all or part of a group of documents belonging to a subset of selected documents and / or information created by the operator, the contents of the cluster can be easily grasped and the operator's own Document classification system that can improve the utility value of classification system, and can gradually collect what kind of contents are included in an arbitrary document set. There is an effect that a method is obtained.

  Further, according to the invention of claim 37, when a plurality of topics and meanings are included in one document, it is classified into a category limited to a specific topic or meaning, or the user's intention Therefore, the user can understand the classification category well. Further, since the position of the divided document in the pre-division document (affiliation document) is also shown, there is an effect that the user can obtain a document classification method that can efficiently read the portion to be read in the document group. .

  According to a thirty-eighth aspect of the invention, by recording a program that causes a computer to execute the method according to any one of the twenty-fourth to thirty-seventh aspects, the program can be read by a machine. The recording medium capable of realizing the operations of Items 24 to 37 by a computer is obtained.

  Exemplary embodiments of a document processing device, a document classification device, a document processing method, a document classification method, and a computer-readable recording medium storing a program for causing a computer to execute the methods according to the present invention are described below with reference to the accompanying drawings. The embodiment will be described in detail.

[Embodiment 1]
First, the hardware configuration of the entire information processing system constituting the document processing apparatus according to Embodiment 1 of the present invention will be described. FIG. 1 is an explanatory diagram showing the hardware configuration of the entire information processing system constituting the document processing apparatus according to the first embodiment.

  In FIG. 1, the information processing system constituting the document processing apparatus according to the first embodiment is configured by a server / client system. That is, the server 101 and a plurality of clients 102 are connected by the network 103. The client 102 generates processing data such as classification data, instructs the server 101, displays processing results such as classification results, and the like. On the other hand, in accordance with an instruction from the client 102, the server 101 performs processing such as document (text) classification by enormous numerical operations, and sends the processing result to the client 102.

  In the case of classification processing, more specifically, the server 101 performs text classification processing (pre-processing and clustering processing), and the client 102 performs classification data generation, processing execution instruction, text classification result display, and the like. It is. As described above, the processing in the server 101 is divided into “pre-processing” and “classification processing”, and the processing becomes very heavy depending on data. Accordingly, in the server 101, the manager process creates and manages a process acceptance list so that only one “pre-process” and “classification process” are performed.

  Further, data exchange between the server 101 and the client 102 uses a method called file sharing. That is, by exchanging data by creating a file used for processing such as classification processing in a shared folder on the server 101. Therefore, the client 102 can use the shared folder of the server 101 by sharing the network.

  Next, the hardware configuration of the server 101 and the client 102 will be described. FIG. 2 is an explanatory diagram showing the hardware configuration of the server 101 in the information processing system constituting the document processing apparatus according to the first embodiment. As the server 101, for example, a workstation (WS) is used.

  In FIG. 2, 201 is a CPU that controls the entire server 101, 202 is a ROM that stores a boot program and the like, 203 is a RAM 203 that is used as a work area for the CPU 201, and 204 is a network 103 via a communication line 205. Reference numeral 206 denotes an interface (I / F) connected to the network 103 and serving as an internal interface, and a disk device 206 stores data. Reference numeral 200 denotes a bus for connecting the above-described units.

  In addition, a display 208 that displays document information, image information, function information, and the like, a keyboard 209 and a mouse 210 for inputting data, and the like may be similarly connected. Further, the disk device 206 is provided with a shared folder 207 for exchanging data with the client 102.

  FIG. 3 is an explanatory diagram showing a hardware configuration of the client 102 in the information processing system constituting the document processing apparatus according to the first embodiment. As the client 102, for example, a personal computer (PC) or the like is used.

  In FIG. 3, 301 is a CPU that controls the entire system, 302 is a ROM that stores a boot program, 303 is a RAM that is used as a work area of the CPU 301, and 304 is an HD (hard disk) 305 according to the control of the CPU 301. HDD (Hard Disk Drive) for controlling the reading / writing of data with respect to the HDD, 305 for storing the data written under the control of the HDD 304, and 306 for reading / writing the data with respect to the FD (floppy disk) 307 according to the control of the CPU 301. FDD (floppy disk drive) for controlling writing, 307 for a removable FD for storing data written under the control of the FDD 306, and 308 for a display for displaying documents, images, function information, etc. That.

  Further, reference numeral 309 is connected to the network 103 via the communication line 310, and an interface (I / F) that controls an internal interface with the network 103, and 311 has keys for inputting characters, numerical values, various instructions, and the like. 312 has a mouse for moving the cursor, selecting a range, pressing an icon or button displayed on the display screen, moving a window, changing the size, etc., and 313 has an OCR (Optical Character Reader) function. 314 indicates a scanner for optically reading the image, 314 indicates a printer for printing the contents of data including the classification result, and 315 indicates a bus for connecting the above-described units. The HD 305 stores application software 316 such as word processing software.

  Next, a functional configuration of the document processing apparatus according to the first embodiment will be described. FIG. 4 is a block diagram functionally showing the configuration of the document processing apparatus according to the first embodiment. In FIG. 4, the document processing apparatus includes an input unit 401, a document storage unit 402, a selection unit 403, a feature extraction unit 404, a processing processing unit 405, and an output unit 406.

  The input unit 401, the document storage unit 402, the selection unit 403, the feature extraction unit 404, the processing unit 405, and the output unit 406 are recorded on a recording medium such as the ROM 202 or 302, the RAM 203 or 303, the disk device 306, or the hard disk 316. The function of each unit is realized by the CPU 201 or 301 executing command processing according to the commands described in the program.

  The input unit 401 inputs document data. For example, the I / O that can obtain a document or a document group via the keyboard 209 or 311, the scanner 313, the scanner 313 having an OCR function, or the network 103. F204 or 309. In addition to the above, the input unit 401 includes all of them as long as it can acquire document data. For example, when document data is stored in a database, the document data is also input when the medium on which the database is recorded is incorporated into the document processing apparatus according to the first embodiment.

  Here, a document is a collection of one or more sentences written in a natural language, and is composed of characters, character strings, numerical values, and the like, and a meaningful group is defined as one document. A group of documents is a document group.

  A document consists of one or more items. An item is composed of an item name and an item value. The item name is a label indicating the content of the item, and may or may not be included in the document. The item value is the actual content of the item. FIG. 5 is an explanatory diagram showing the relationship between item names and item values of the document processing apparatus according to the first embodiment. For example, one patent gazette is one document, and when the patent gazette is expressed by item names and item values, it is as shown in FIG.

  A document or document group acquired by the input unit 401 is assigned a unique document ID to each document and stored in the document storage unit 402. FIG. 6 is an explanatory diagram showing the data structure of a document stored in the document storage unit 402 of the document processing apparatus according to the first embodiment. Each item name or item value is stored in a cell of the document storage unit 402, that is, one storage unit.

  In FIG. 6, one cell includes three storage areas, and the first storage area 601 stores the position (address) of the next cell on the document storage unit 402. The second storage area 602 stores cell attribute values.

  For example, “0” is “empty”, “1” is “numerical value”, “2” is a character string, etc. as cell attribute values. The third storage area 603 stores the actual contents of the cell, that is, the start position of the area in which the item name or item value is stored.

  Rearranging the order of cells and adding / deleting cells can be easily realized by changing the position of the next cell stored in the first storage area 601. In addition, since the actual contents of the cell are stored in a different area different from the cell structure, for example, if the result of changing an item does not fit in the reserved area, the cell structure There is no influence on itself, and it is only necessary to secure a separate large area, store the item value therein, and change the top position of the storage area 603 stored third.

  FIG. 7 is an explanatory diagram showing another data structure of a document stored in the document storage unit 402 of the document processing apparatus according to the first embodiment. In FIG. 7, one cell uses two storage areas. In the first storage area 701, cell attribute values are stored. The second storage area 702 stores the actual contents of the cell, that is, the start position of the area in which the item name or item value is stored.

  The next cell is stored in the next storage area on the document storage unit 402. In this data structure, when the rearrangement of the cell order and the addition / deletion of the cell occur, it is necessary to move the stored contents.

  The document storage unit 402 is normally composed of a semiconductor memory capable of handling information at high speed, but may include an auxiliary storage device composed of a magnetic disk or an optical disk.

  The document or document group stored in the document storage unit 402 is displayed by the output unit 406. In the first embodiment, the output unit 406 includes a CRT display, a liquid crystal display, or the like. The output unit 406 reads the contents of the document or document group stored in the document storage unit 402 by sequentially tracing the cells, and displays or prints the data in a table format.

  The output unit 406 may include a graph drawing unit 407 that draws a graph based on data displayed or printed in a table format. The graph drawing unit 407 reads the contents of the area set by the user with respect to the item value of the document or document group stored in the document storage unit 402, and graphs such as a bar graph, a pie graph, and a line graph according to the user's instruction Draw and display or print.

  The output unit 406 also performs display related to operations by the input unit 401, such as an operation menu, a mouse pointer, and a cursor. Further, a printing apparatus such as a printer for printing the processing result may be included.

  The selection unit 403 reads the data of the area selected on the display of the output unit 406 from the document storage unit 402 in accordance with an operator instruction from the input unit 401, and sends the data to the feature extraction unit 404. The selection method of the selection part 403 is demonstrated using FIGS. 8-10.

  FIG. 8 to FIG. 10 are explanatory diagrams showing examples of screen display by the output unit 406 of the document processing apparatus according to the first embodiment, specifically, examples of screen display on which the content of the failure state of the car is displayed. . In FIG. 8, the screen display includes a “number” column 801 indicating a document ID number, a “reception date” column 802 indicating a date when failure information is received, and a “sales office” column 803 indicating a sales office receiving the failure information. A “vehicle type” column 804 indicating the type of the vehicle that is the target of the failure information, a “year” column 805 indicating the year of the vehicle that is the target of the failure information, and a “content” column 806 indicating the content of the failure status Is displayed.

  In FIG. 9, a selection area 901 is a part surrounded by a rectangle and the display color is changed. Similarly, in FIG. 10, the selection area 1001 is surrounded by a rectangle and a part whose display color is changed. It is.

  The area selected by the selection unit 403 may be a part of a column on the screen as shown in FIG. 9, or when an item name is selected as shown in FIG. All the item values to which it belongs may be selected. In the first embodiment, only an area having a character string attribute can be selected.

  Next, the contents of extraction processing performed by the feature extraction unit 404 will be described. For the item value selected by the selection unit 403, the feature extraction unit 404 extracts the feature of the item value. FIG. 11 is an explanatory diagram showing a list of contents of extraction processing performed by the feature extraction unit 404 of the document processing apparatus according to the first embodiment.

  In FIG. 11, the extraction process includes a word included in the target character string, the number of words in the word, the number of characters in the word, and the number of appearances of each word. . . Etc. These extraction processes are realized by using a natural language processing technique such as a morphological analysis technique or a syntax analysis technique that is generally used, such as a regular speech synthesizer or an automatic translation apparatus.

  Next, the contents of the processing performed by the processing unit 405 will be described. The processing unit 405 performs processing on the feature amount extracted by the feature extraction unit 404. FIG. 12 is an explanatory diagram showing a list of contents of the processing performed by the processing unit 405 of the document processing apparatus according to the first embodiment.

  The processing includes “classification process” for classifying the same feature quantity, “search process” for searching for a predetermined feature quantity, “sorting process” for sorting by feature content, and representative values of feature quantities "Representative value extraction process" to extract the maximum value of the feature quantity, "Maximum value extraction process" to extract the maximum value of the feature quantity, "Minimum value extraction process" to extract the minimum value of the feature quantity, "Arithmetic" Processing ".

  The combination of the content of the feature amount extraction process performed by the feature extraction unit 404 and the content of the feature amount extraction process performed by the processing unit 405 can be selected by each operator. Alternatively, a combination that is highly effective may be set in advance, and the set combination may be provided to the operator.

  The processing result processed by the processing unit 405 is held in a processing result holding unit 408 in the processing unit 405. The processing result held in the processing result holding unit 408 is output by the output unit 406. The output unit 406 reads the contents from the processing result holding unit 408 and performs image display and print output.

  Here, the case where the number of appearances of each word included in the item value is selected as the feature (amount) extracted by the feature extraction unit 404 and the classification process is selected as the processing performed by the processing unit 405 will be described. To do.

  Generally, when there are two documents and the appearance frequencies of the words constituting the two documents are equal, it can be considered that the meanings of the two documents are similar. In other words, the number of occurrences of a word in a document can be considered as a feature quantity closely related to the meaning of the document. Therefore, when a plurality of documents are classified using the number of occurrences of the word as a feature amount, it can be considered that documents having similar meanings belong to each classification category.

  One or a plurality of item values selected by the selection unit 403 are subjected to natural language analysis such as morphological analysis for each item value by an analysis unit 409 included in the feature extraction unit 404, and divided into words. Each word is also given part-of-speech information of the word. Among the appearing words, a unique word ID is assigned to a noun, and the number of appearances for each word ID for one item value and all the item values selected by the selection unit 403 is counted.

The feature vector generation unit 410 included in the feature extraction unit 404 generates an item value feature vector indicating the feature (amount) of each item value based on the counted number of appearances. For example, the item value selected by the selection unit 403 is
“Noisy”
"Paint changes color"
"Overheating occurs"
"Painting comes off"
"Battery goes up"
"The exhaust is black"
In this case, the feature vector of each item is as shown in FIG. FIG. 14 shows a word and the number of appearances for each word ID.

That is,
“Noisy”: {1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
"Paint changes color": {0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0}
"Overheating occurs": {0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0}
"The paint peels off": {0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0}
“Battery goes up”: {0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0}
“The exhaust is black”: {0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1}
Is obtained.

  The feature vector of this item value is sent to the processing unit 405 as an output from the feature extraction unit 404. In the processing unit 405, classification processing is performed using the feature vector of the item value. The classification process is first obtained by calculating the distance between the vectors. For example, an inner product can be used as a measure of the distance.

  After calculating the distance, the process of collecting the objects that are close to each other is performed. For this process, for example, a K-means method for classifying a set of vectors into a set of K vectors according to the distance can be used. When the classification of each vector is completed, the number of which classification belongs to the item value corresponding to the vector, that is, the cluster number and the document ID corresponding to the item value are given, and the output of the processing unit 405 The output unit 406 performs output display.

  FIG. 15 shows a display example of a screen displaying the cluster number 1501. The documents having the same cluster number, for example, the documents having the numbers “1” and “6” have the cluster number “5”, indicating that both belong to the same classification.

  Next, a case where a cross table is output in the invention of claim 2 will be described. After reading a group of documents to be analyzed by the input unit 401, the operator classifies the item name to be subjected to classification processing, the item name on the horizontal or vertical axis of the cross table, and the number of classifications to be performed. Specify the number.

  FIG. 16 is an instruction screen for creating a cross table. In FIG. 16, the instruction screen 1600 includes a processing target item name input field 1601, an item name input field 1602 serving as an axis, a vertical axis designation button 1603, a horizontal axis designation button 1604, and a classification number input field 1605. Is done.

  In the processing target item name input field 1601, an item name to be processed is input. An item name can be input by inputting from the keyboard 209 or the like, or by displaying an item candidate to be processed and selecting it from the mouse 210 or the like. In addition, an item name to be an axis is input in an item name input field 1602 to be an axis. The input method is the same as the input method to the processing target item name input field 1601.

  The vertical axis designation button 1603 and the horizontal axis designation button 1604 are buttons for designating whether to display items on the vertical axis on the vertical axis or on the horizontal axis. In the number-of-classification input field 1605, the number of classifications is input. As an input method, the number of classifications may be input by inputting numbers from the keyboard 209 or the like, or by displaying candidates for the number of classifications and selecting them from the mouse 210 or the like.

  In FIG. 16, “content” is checked in the processing target item name input field 1601, “car type” is checked in the item name input field 1602 serving as the axis, and the horizontal axis designation button 1604 is checked, and the classification number input field 1605 is checked. “50” is input to the field, and based on the “content” in the document group, it is classified as “50 (pieces)”, and an instruction to display “vehicle type” on the horizontal axis of the cross table is given. You can see that

  By instructing creation of a cross table, classification processing is executed, and the result is displayed in a cross table. 17 and 18 are diagrams showing a cross table on which the result of the classification process is displayed. In FIG. 17, the cross table 1700 includes “cluster 1”, “cluster 2”. . . "ABC1600", "ABC1800", etc. indicating the vehicle type on the horizontal axis. . . Is displayed.

  The vertical axis of the table, that is, each row corresponds to a cluster generated by the classification process. The first column of each line contains a character string indicating the cluster number as a default value at the end of the classification process. On the horizontal axis of the table, that is, in each column, the character strings included in the item “car type” of the document group are displayed without duplication. In each cell of the row “cluster 1”, the number of documents classified as cluster 1 whose item “car type” matches the car type in the column is displayed.

  Here, instead of displaying the number, the size of the number may be expressed by the shade of the cell color or the area where the cell is filled. In the rightmost column and the lowermost column of the table, the totals of the corresponding rows and columns are displayed.

  In FIG. 18, when the mouse pointer 1800 is moved to a certain cell in the cross table 1700 and the mouse button of the mouse 210 is pressed or the cursor is moved by the cursor key operation of the keyboard 209 and the specific key is pressed, the vicinity of the cell is displayed. By displaying the content display screen 1801, the item “content” of the corresponding document is displayed.

  The content display screen 1801 displays the number of data in a cell, display items, cell information, and the contents of display items in each data. In the cell designated by the mouse pointer 1800, the number of data is “4”, the display item is “content”, the cell information is “ABC2000-cluster 1”, and the content of the display item is “content”. “The exhaust is black, the exhaust is black,” is displayed. Thus, the contents of the cell can be recognized by a simple operation of moving the mouse pointer to a desired cell and pressing the mouse button.

  The items displayed on the content display screen 1801 can be changed by a setting operation, and all items can be displayed, or items can be selected and displayed.

  The first column of each line contains a character string indicating the cluster number as a default value at the end of the classification process, but this column can be rewritten by the operator. For example, after confirming the contents of the cell by the above operation, “cluster 1” can be rewritten as “exhaust problem”. This makes it easier to understand the information content.

  Further, instead of entering a character string indicating a cluster number as a default value at the end of classification, it is possible to extract a character string indicating the characteristics of the cluster and put it in a cell. For example, it is realized by extracting a sentence or word that appears most frequently from the item “content” of documents included in the cluster 1.

  In FIG. 18, words such as “exhaust is black” or “exhaust” are entered in cluster 1. In this way, the operator can not only grasp the distribution state of the entire document by a simple operation, but can also know the contents of individual documents in detail as necessary.

  Next, the details of the configuration of the output unit 406 for creating a cross table will be described. FIG. 19 is a block diagram illustrating a detailed configuration of the output unit 406 of the document processing apparatus according to the first embodiment. In addition to the graph drawing unit 407, the output unit 406 includes an item value selection unit 1901 and a totaling unit 1902. The totaling unit further includes a table holding unit 1903 having a storage area corresponding to the content to be actually displayed. Yes.

  The item value selection unit 1901 sequentially reads item values from the document data stored in the document storage unit 402 for the item name (axis item name) designated as one axis of the cross table by the operator, Create a set of missing field values. The totaling unit 1902 adds the numerical values to the area corresponding to the item value of the table holding unit 1903 and totals the documents.

  Next, the procedure for outputting the cross table will be described. FIG. 20 is a flowchart showing a cross table output procedure of the document processing apparatus according to the first embodiment. In the flowchart of FIG. 20, first, the contents of the table holding unit 1903 are initialized prior to aggregation (step S2001).

  Next, the item value created by the item value setting unit 1901 is applied to the portion corresponding to the item value label in the table (step S2002), and the character string representing the cluster number is applied to the portion corresponding to the cluster number (step S2002). S2003).

  Next, for the document ID corresponding to the item value held in the processing result holding unit 408, the corresponding document stored in the document storage unit 402 is referred to, and the item value corresponding to the axis item name is determined ( Step S2004). Thereafter, 1 is added to the contents of the corresponding area of the table holding unit 1903 (step S2005).

  It is determined whether or not the above processing has been performed for all item values (step S2006). If the above processing has not been performed for all item values (No in step S2006), the process proceeds to step S2004, and steps S2004 to S2006. Repeat the process.

  In step S2006, when the above-described processing has been performed for all item values (Yes in step S2006), the total number of rows to be displayed in the rightmost column is calculated (step S2007) and displayed together in the bottom row. The sum of the fields for the calculation is calculated (step S2008).

  Thereafter, the tables configured in the table holding unit 1903 are sequentially read and output (step S2009), and all the processes are terminated.

  Note that the data output from the processing unit 405 may be sent to the document storage unit 402 and stored together with other data in the document storage unit 402. Data output from the processing unit 405 stored in the document storage unit 402 can be displayed as a new column in the table by the output unit 406. Alternatively, an existing column of the table may be deleted and overwritten on the deleted column.

  In this configuration, the data output from the processing unit 405, which is the result of the processing, can be handled in the document storage unit 402 on an equal basis with other data that is not the target of the current processing, and the subsequent analysis It is possible to select the target of the processing without distinguishing whether it was present in the original input data or generated by the processing during the analysis operation.

  Therefore, the processing target and the processing content can be selected flexibly according to the nature of the data and the content of the information analysis operation to be performed, so that a variety of advanced information analysis operations can be performed.

  Further, as input data to the processing unit 405, not only data output from the feature extraction unit 404 but also data selected by the selection unit 403 can be included. As a result, it is possible to perform further processing on data that does not require character string feature extraction and on the numerical value of the processing result, thereby enabling more diverse and advanced information analysis.

  FIGS. 21 to 24 are explanatory diagrams showing another example of screen display by the output unit 406 of the document processing apparatus according to the first embodiment. In FIG. 21, in addition to “number”, “reception date”, “sales office”, “vehicle type”, “year”, “content”, “cluster number” 2101 which is a result obtained by the classification process is displayed. Has been.

  Further, in FIG. 21, the “cluster number” 2101 is selected by the selection unit 403, and the data related to the “cluster number” 2101 is highlighted. Using the selected “cluster number” 2101 as a key, the processing unit 405 instructs the rearrangement process to be performed.

  FIG. 22 shows the result of the rearrangement process in accordance with the rearrangement process instruction. In FIG. 22, rearrangement is made so that the “cluster number” of “1” is collected and displayed, and subsequently, rearrangement is made so that the “cluster number” of “2” is gathered and displayed. It is done.

  Specifically, the “cluster number” is “1” and the “number” is “2”, “11”, “15”, “23”, “35”, “54”, “63”, “73”. , “82”, “cluster number” is “2”, and “number” is “14”, “18”, “22”, “27”, “37”,. . . It can be seen that is displayed.

  Next, in the column of the item “vehicle type”, the one whose “cluster number” belongs to “1” is selected. FIG. 23 shows that the item “vehicle type” in which the “cluster number” belongs to “1” is selected, and the selection area 2301 is highlighted. As described above, the “cluster numbers” have already been rearranged, and the objects belonging to the same cluster are displayed together, so that they can be easily selected as continuous areas on the screen.

  Next, FIG. 24 shows a bar graph of the occurrence frequency of the vehicle type in the selection area 2301. In FIG. 24, nine documents with “cluster number” “1” selected in the selection area 2301 are selected in the bar graph display area 2401, and the nine documents are displayed as a bar graph according to vehicle type. .

  In this way, the processing target can be selected flexibly and easily, various processing can be performed on the selected target, and the processing result can be the target of the next processing. Advanced information analysis work becomes possible.

  In this way, examples of performing various types of processing after extracting feature values of character strings such as classification and performing processing using the feature amounts have been shown, but performing various types of processing in advance You may be able to.

  For example, after selecting the item “vehicle type” and rearranging it using this as a key, classification processing may be performed on a certain vehicle type collected, for example, “ABC1600”. In addition, when a document input by the input unit 401 includes an error such as a typographical character, before extracting a feature amount of a character string such as a classification and performing processing using the feature amount, for example, a character By performing column search / replacement processing, typographical errors can be corrected in a lump, and data can be arranged so that more favorable results can be obtained.

  FIG. 25 is a block diagram showing a detailed configuration of the document storage unit 402 of the document processing apparatus according to the first embodiment. 25, the document storage unit 402 includes a setting value storage unit 2501 and a setting value transmission / reception unit 2502. The setting value storage unit 2501 stores various setting values such as the classification information storage unit 2503 such as the number of classifications for classifying documents, that is, information on setting values necessary for the operation of the document processing apparatus. It has. Thereby, the information regarding the set value can be stored together with the document information.

  Also, the setting value transmission / reception unit 2502 transmits information on the setting value stored by the setting value storage unit 2501 to another information processing apparatus. In addition, the setting value transmission / reception unit 2502 receives information on setting values from other information processing apparatuses. Information on the setting value received by the setting value transmission / reception unit 2502 is stored in the setting value storage unit 2501.

  Information regarding the stored setting value is read at the same time when the document is read again later and stored in the setting value storage unit 2501. Information on the set value can be referred to by an operator performing a predetermined operation, or can be reused in subsequent processing. As a result, it becomes possible to store and manage information related to the setting values together with the document, so that loss of information related to the setting values can be prevented and suitable setting values can be reused later.

  26 to 28 are explanatory diagrams illustrating another example of screen display by the output unit 406 of the document processing apparatus according to the first embodiment. In FIG. 26, first, the “content” that is an object to be classified by the operator is selected on the display screen. As a result, the selection area 2601 is highlighted. Next, when a classification processing button 2603 is selected from the menu bar 2603, an inquiry screen 2604 about the number of classifications necessary for the classification processing, that is, how many the objects are classified is displayed.

  When the operator inputs the number of classifications on the inquiry screen 2604, information regarding the number of classifications is stored in the document storage unit 402. In FIG. 26, “50” is input as the number of classifications.

  Thereafter, when the operator completes the information analysis work and presses a save button (not shown) that pops up when the file button 2605 of the menu bar 2603 is selected, the file name designated by the operator is displayed by the document storage unit 402. Assigned and stored together with document information and classification results.

  In FIG. 27, when a mouse pointer 2702 is moved to a column 2701 for displaying a classification result and a mouse button is pressed, a classification information display screen 2703 that displays information related to classification and information related to classification setting values used for performing the classification. Is displayed. Thereby, the association of the used setting values can be easily grasped.

  The classification information display screen 2703 displays, for example, “classification date” indicating information regarding the date and time when classification is performed as information regarding classification, “number of classification targets” indicating information regarding the number of documents targeted for classification, and the like. Further, as the information regarding the classification setting value, “classification number” indicating how many are classified, “classification part of speech” indicating the classification based on which part of speech, and the like are displayed.

  A new table is created each time the classification process is executed. FIG. 28 shows a state where the classification result 1 is displayed after the classification result 1 is obtained and the classification result 2 is displayed. In order to display the classification result 1 again, the mouse pointer is moved to the selection area 2801 on the label at the lower left of the screen, and the mouse button is pressed. Thereby, the classification result 1 is displayed again. Thereafter, when the classification result 2 is displayed again, the same operation can be performed.

  Also, in FIG. 28, information regarding the set value used for executing each classification process is displayed in a predetermined display area 2802 of the corresponding table. The display area 2802 can be displayed so as not to hide the display of the classification result, and the display position can be moved. Thereby, the association between the classification result and the setting value used for it can be easily grasped.

  Next, a series of document processing procedures of the document processing apparatus according to the first embodiment will be described. FIG. 29 is a flowchart showing a series of document processing procedures of the document processing apparatus according to the first embodiment.

  In the flowchart of FIG. 29, first, it is determined whether or not document data is input to the document processing apparatus (step S2901). When the document data is input after waiting for the input of the document data (Yes at step S2901), the input document data is stored (step S2902). Note that each of steps S2901 and S2902 may be performed independently of other steps each time a document is input.

  Next, it is determined whether all or a part of the stored document data has been selected (step S2903). If all or part of the document data is selected and selected (Yes in step S2903), data relating to the character string characteristics of all or part of the selected document data is extracted. Perform (step S2904).

  After that, in step S2904, predetermined processing such as classification processing is performed based on the data relating to the extracted character string characteristics (step S2905). Subsequently, an output process such as expanding the data processed in step S2905 into a table format is performed (step S2906).

  Further, the data processed in step S2905 is stored in association with the original document data (step S2907). In addition, data related to the content of the machining process such as a set value of the machining process is also stored (step S2908).

  Thereafter, it is determined whether or not all or part of the data processed in step S2905 has been selected (step S2908). If not selected (NO in step S2909), the process proceeds to step S2904, and thereafter, step The processing from S2904 to S2909 is repeated. On the other hand, if all or part of the processed data is selected in step S2909 (Yes in step S2909), all the processes are terminated.

  The document processing method described in the first embodiment is realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. This program is recorded on a computer-readable recording medium such as a hard disk, floppy disk, CD-ROM, MO, and DVD, and is executed by being read from the recording medium by the computer. The program can be distributed through the recording medium or as a transmission medium via a network such as the Internet.

  Next, information classification apparatuses according to Embodiments 2 to 6 will be described. In Embodiments 2 to 6 to be described below, based on the interpretation that it contains a lot of noise as described above, it is positioned as topic (content) extraction from a single document set, and the document Arbitrary document sets can be created by providing a means to execute classification classification while changing the parameters for classification (target document set, number of clusters, similarity measure, stop word, etc.), and to retain and integrate the results. The contents of what are included in the are collected gradually.

[Embodiment 2]
The information processing system constituting the document classification apparatus according to the second embodiment of the present invention is the same as the information processing system according to the first embodiment as shown in FIG. The hardware configurations of the server 101 and the client 102 are the same as those in the first embodiment as shown in FIGS.

  Next, a functional configuration of the document classification apparatus according to the second embodiment will be described. FIG. 30 is a block diagram functionally showing the configuration of the document classification apparatus according to the second embodiment.

  In the block diagram of FIG. 30, the document classification apparatus includes an input unit 3001, a language analysis unit 3002, a vector generation unit 3003, a classification unit 3004, a classification parameter instruction unit 3005, a classification result storage unit 3006, and a cluster feature. The display unit 3007 includes a cluster feature calculation unit 3008, a classification system storage unit 3009, a cluster selection instruction unit 3010, and a classification system browsing operation unit 3011.

  Input unit 3001, language analysis unit 3002, vector generation unit 3003, classification unit 3004, classification parameter instruction unit 3005, classification result storage unit 3006, cluster feature display unit 3007, cluster feature calculation unit 3008, classification system storage unit 3009, cluster selection The instruction unit 3010 and the classification system browsing operation unit 3011 are processed by the CPU 201 or 301 according to instructions described in a program recorded in a recording medium such as the ROM 202 or 302, the RAM 203 or 303, or the disk device 306 or the hard disk 316. By executing the above, the function of each part is realized.

  Here, the input unit 3001 inputs document data. For example, a document or a document group can be obtained via the keyboard 209 or 311, the scanner 313, the scanner 313 having the OCR function, or the network 103. I / F 204 or 309 that can be used.

  In addition to the above, the input unit 3001 includes all of them as long as it can acquire document data. For example, when the document data is stored in a database, the document data is also input when the medium in which the database is recorded is incorporated in the document classification apparatus according to the present embodiment.

  The language analysis unit 3002 analyzes the document data input by the input unit 3001 and obtains language analysis information. The vector generation unit 3003 is based on the language analysis information obtained by the language analysis unit 3002. A document feature vector for the document data is generated.

  The classification unit 3004 classifies the document based on the similarity between the document feature vectors generated by the vector generation unit 3003, and generates a subset of the document. The classification parameter instruction unit 3005 includes a classification parameter For example, the keyboard 209 or 311, the mouse 210 or 312, or the I / F 204 or 309 that can obtain the instruction information via the network 103.

  The classification result storage unit 3006 stores the result of classification by the classification unit 3004, that is, information related to a subset of classified documents. The cluster feature display unit 3007 displays the cluster feature calculated by the cluster feature calculation unit 3008.

  The cluster feature calculation unit 3008 calculates a cluster feature that is a feature of a subset of documents generated by the classification unit 3004. The classification system storage unit 3009 stores the cluster features calculated by the cluster feature calculation unit 3008 as components of the classification system. The classification system storage unit 3009 stores a subset of documents selected by the cluster selection instruction unit 3010 as a component of the classification system. That is, all the documents belonging to the cluster selected by the cluster selection instruction unit 3010 or a part of the belonging documents are stored as components of the classification system.

  The cluster selection instruction unit 3010 is for selecting a desired cluster from the plurality of cluster features displayed by the cluster display unit 3007. The cluster selection instruction unit 3010 selects a desired subset from the document subsets generated by the classification unit 3004. In addition, the classification system browsing operation unit 3011 performs a browsing operation when it is desired to browse data stored in the classification system storage unit 3009.

  Next, a suitable example in which it is important to extract topics (contents) contained in a document set will be explained using a specific example, assuming an analysis situation of free description answers obtained by questionnaire surveys, etc. .

  In recent years, for example, thousands to tens of thousands of free description answers can be collected in a short time via the Internet or the like, and a large amount of text information can be collected using such a function.

  As an example of collecting a large amount of text information obtained from a questionnaire survey, if a document is a single answer description that is answered in response to the question “Please waste by networking the office”, a document set (cluster ) Is a set of responses for each case.

  Here, as one of the needs, the operator (questionnaire analyst) may know what kind of opinions (topics) are included in the opinion set (document set) and want to grasp the outline of the opinion. is there. In order to satisfy such needs, topic extraction is realized by a group (classification) of similar opinions, and what kind of opinions are included in the questionnaire result is extracted.

  The document classification is typically roughly divided into the following three steps. In the first step, for each document (opinion) input by the input unit 3001, the language analysis unit 3002 extracts a word (or a specific continuous character string) included in each document. At this time, for example, a language analysis algorithm such as a morpheme trace is used.

  In the second step, a matrix of “word” × “document” is generated in which the extracted word is a column, each document is a row, and the element is an appearance frequency of the word. If you use a language analysis tool that has a general morphological analysis function and a syntax analysis function, in addition to word extraction, you can simultaneously acquire word part-of-speech information, compound words (phrases), syntax information, etc. Can be taken into account when generating the word × document matrix.

  The vector generation unit 3003 represents each document as a vector in a multidimensional space composed of words based on the matrix of “word” × “document”. This includes the following methods, and all methods are implemented in the present embodiment.

(1) A method of using matrix column components as they are,
(2) A method of weighting values in consideration of the length of each document (number of characters, number of pages, etc.) and the frequency of occurrence of each word in the document set of the entire classification target,
(3) Calculate an inner product matrix between documents from the above matrix and apply singular value decomposition (for example, using factor analysis, principal component analysis, quantification theory type 3 etc.) to this potential A method of constructing a semantic space,
Etc.

  In addition, "Representating Documents Using an Explicit Model of Their Similarities (Author: Brian T. Bartell, Garrison W. Cottrell, and Richard K. Belew, paper name: Journal of the American Society for Information Science, Society name: the American Society "For Information Science, Page: 254-271, Vol. 46 No. 4, Publication year: 1995)", the conversion method to the above latent semantic space is generalized. Generated from reference information to other documents Some models derive expression space conversion functions for projecting documents and words to a space that reflects these similarities using a matrix with added co-reference information. it can.

  In the third step, the classification unit 3004 classifies the document using the similarity of the document feature vectors. Specifically, classification is performed by applying a chi-square method, a discriminant analysis method, a clustering method, or the like to the classification target data.

  The similarity may be an inner product, cosine, Euclidean distance, Mahalanobis distance, or the like, and any method may be used in the present embodiment.

  Various clustering algorithms are also known. Clustering can be broadly divided into hierarchical clustering and non-hierarchical clustering, but any method may be used in the present embodiment.

  The classification parameter instruction unit 3005 instructs a classification parameter for the classification unit 3004 to classify the document feature vector. The classification unit 3004 classifies the document feature vector held therein according to the classification parameter instructed by the classification parameter instruction unit 3005.

  In this manner, when the first document classification is completed by executing the processes of the first step to the third step, the classification result is held in the classification result storage unit 3006.

  Subsequently, the cluster feature calculation unit 3008 calculates a feature indicating what kind of cluster the classification result has been obtained, that is, a class feature. Typically, a document belonging to each cluster or a part of the document is calculated. At this time, the documents are sorted and output based on the similarity to the center of gravity of the cluster.

  In addition, numerical values such as the most frequent word in the cluster, the number of documents belonging to the cluster, and the standard deviation in the cluster representing the degree of variation of the documents in the cluster are calculated as the characteristics of the cluster.

  The feature information of these clusters is calculated so that the operator can understand what kind of cluster the output (displayed) cluster has (what kind of features it has). On the other hand, as long as it shows the characteristics of the cluster, it may be other than the above contents (features).

  The cluster feature calculation unit 3008 also calculates information indicating the relationship between the clusters in addition to the cluster features as described above. In the case of hierarchical clustering, upper or lower clusters are calculated, and in the case of non-hierarchical clustering, adjacent clusters based on the similarity between cluster centroids are calculated.

  Next, the cluster feature display and cluster selection contents by the cluster feature display unit 3007 will be described. FIG. 31 is an explanatory diagram illustrating an example of display on the cluster feature display unit 3007 of the document classification device according to the second embodiment.

  In FIG. 31, the operator can perform operations in units of clusters. Each cluster includes a “cluster ID” column 3101, a “number of members” column 3102, a “frequency word” column 3103, a “document content” column 3104, It consists of items such as “similarity with the center of gravity” column 3105.

  In the “cluster ID” column 3101, a number indicating a cluster ID is assigned as a serial number and displayed. In the “number of members” column 3102, the number of documents belonging to the cluster or a part of the documents is calculated and displayed. Among them, frequently used words are extracted and displayed in the “Frequently Used Words” column 3103. The “document content” column 3104 displays the content of the document, and the “similarity with the center of gravity” column 3105 displays the degree of similarity with the digitized center of gravity. Thereby, an operator's understanding ease improves.

  The operator can grasp the feature of the cluster based on the displayed information (feature amount). Here, if there is even one cluster whose contents (features) can be understood, the operator can select a cluster by the cluster selection instruction unit 3010.

  More specifically, by moving the cursor 3110 to a predetermined position of the displayed cluster, for example, the “cluster ID” column 3101 with the mouse 210 or 312 or the like and clicking at that position, the cluster ID of the cluster ID is displayed. The entire cluster can be selected. Note that not all of the documents belonging to the selected cluster are necessarily selected, and some of the documents may be selected.

  In FIG. 31, the “cluster ID” column 3101 is clicked, whereby the entire cluster is highlighted, indicating that the cluster (cluster ID “1”) has been selected.

  Further, when there is no cluster whose contents can be understood, the operator can reset the classification parameter by the classification parameter instruction unit 3005 and perform classification again.

  Data relating to the cluster ID selected by the cluster selection instruction unit 3010 is transmitted to the classification system storage unit 3009. The classification system storage unit 3009 searches the cluster feature calculation unit 3008 for the feature quantity related to the cluster based on the data related to the cluster ID and stores it.

  Similarly, the classification system storage unit 3009 retrieves the classification result from the classification result storage unit 3006 and stores it. Furthermore, the classification system storage unit 3009 can also store information on comments (for example, “the network maintenance cost is high”) regarding the cluster input by the operator. In this way, by storing information created by the operator as a component of the classification system, the utility value of the classification system is further improved.

  Note that the data stored in the classification system storage unit 3009 can be manually or manually held to view the contents of clusters to be selected / held and the semantic relationship between clusters by providing a separate browsing operation interface. It can be structured and systematized automatically using the similarity between the cluster centroids.

  Next, a series of processing procedures of the document classification device according to the second embodiment will be described. FIG. 32 is a flowchart illustrating a series of processing steps of the document classification device according to the second embodiment. In the flowchart of FIG. 32, first, a document to be classified is input (step S3201).

  Next, the language of the input document is analyzed (step S3202), and a document feature vector is generated based on the analyzed result, that is, the extracted word (step S3203).

  Then, after waiting for a classification parameter instruction, if there is a classification parameter instruction (Yes in step S3204), the document is classified according to the classification parameter instructed (step S3205). Information about the cluster is stored (step S3206).

  Next, the feature of the classified cluster is calculated (step S3207), and the calculated result is displayed (step S3208). It is determined whether or not a cluster has been selected from the displayed clusters (step S3209). If not selected (No in step S3209), the process proceeds to step S3204, and there is an instruction for a classification parameter again. Wait (step S3204).

  On the other hand, if a cluster is selected in step S3209 (Yes in step S3209), a classification system is generated and stored for the selected cluster (step S3210). At this time, information about the cluster input by the operator can also be stored. As a result, the series of processes is completed.

  As described above, according to the document classification device according to the second embodiment, each document is reflected on the semantic relationship between the documents based on the similarity between documents in the document group to be classified. By calculating an expression space conversion function for conversion into a possible expression space and performing document classification in the expression space, document classification that can reflect the operator's intention can be realized.

  Therefore, the cluster can be obtained by the classification unit 3004, and the cluster feature calculation unit 3008 and the classification system storage unit 3009 can use the similarity between the cluster centroids to structure and organize the cluster based on the cluster contents. Can be done.

  Further, using only the clusters selected by the cluster selection instructing unit 3010, it is possible to structure and organize the clusters closer to what the operator intended.

[Embodiment 3]
Now, in addition to the above-described second embodiment, a configuration that further includes a vector storage unit and a vector correction unit as in a third embodiment described below may be adopted.

  The information processing system constituting the document classification apparatus according to the third embodiment is the same as that of the first embodiment as shown in FIG. The hardware configurations of the server 101 and the client 102 are the same as those in the first embodiment as shown in FIGS.

  Next, a functional configuration of the document classification apparatus according to the third embodiment will be described. FIG. 33 is a block diagram functionally showing the structure of the document classification device according to the third embodiment of the present invention. 33, the same components as those in FIG. 30 of the second embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  In the block diagram of FIG. 33, the document classification apparatus includes an input unit 3001, a language analysis unit 3002, a vector generation unit 3003, a classification unit 3004, a classification parameter instruction unit 3005, a classification result storage unit 3006, a cluster feature display unit 3007, a cluster feature. In addition to the calculation unit 3008, the classification system storage unit 3009, the cluster selection instruction unit 3010, the classification system browsing operation unit 3011, a vector storage unit 3301 and a vector correction unit 3302 are included.

  The vector storage unit 3301 stores the document feature vector generated by the vector generation unit 3003. Further, the vector correction unit 3302 corrects the document feature vector stored in the document feature vector storage unit 3301 so that the document feature vector of the document belonging to the subset selected by the cluster selection instruction unit 3010 is removed. To do.

  The classifying unit 3004 classifies the document based on the document feature vector corrected by the vector correcting unit 3302.

  The vector storage unit 3301 and the vector correction unit 3302 are instructed by the CPU 201 or 301 according to instructions described in a program recorded on a recording medium such as the ROM 202 or 302, the RAM 203 or 303, or the disk device 306 or the hard disk 316. By executing the processing, the function of each unit is realized.

  The document feature vector (column vector) / word (word feature) vector (row vector) generated by the vector generation unit 3003 is stored in the vector storage unit 3301. This is for securing a document feature vector to be used in the subsequent classification execution.

  The vector correction unit 3302 deletes all or some of the documents belonging to the cluster selected by the cluster selection instruction unit 3010 so that these documents will be removed from the next time. The deleted document feature vector is stored in the vector storage unit 3301.

  As a result, the vector data stored in the vector storage unit 3301 excluding the document belonging to the selected cluster (or a part thereof designated by the operator) column vector is classified as the next and subsequent classes. Data used when executed.

  Next, a series of processing procedures of the document classification device according to the third embodiment will be described. FIG. 34 is a flowchart showing a series of processing procedures of the document classification device according to the third embodiment. In the flowchart of FIG. 2, first, a document to be classified is input (step S3401).

  Next, the language of the input document is analyzed (step S3402), a document feature vector is generated based on the analyzed result, that is, the extracted word (step S3403), and the generated document feature vector is Stored (step S3404).

  After that, when there is an instruction for the classification parameter, and when there is an instruction for the classification parameter (Yes at Step S3405), the document is classified according to the specified classification parameter (Step S3406), and as a result, that is, Information about the cluster is stored (step S3407).

  Next, the feature of the classified cluster is calculated (step S3408), and the calculated result is displayed (step S3409). It is determined whether or not a cluster has been selected from the displayed clusters (step S3410). If not selected (No in step S3410), the process proceeds to step S3405, and there is an instruction for a classification parameter again. Wait (step S3405).

  On the other hand, if a cluster is selected in step S3410 (Yes in step S3410), a classification system is generated and stored for the selected cluster (step 3411). At this time, information about the cluster input by the operator can also be stored. Thereafter, it is determined whether or not there is an instruction to perform repeated processing (step S3412).

  If there is an instruction to repeat processing in step S3412 (Yes in step S3412), the document feature vector is corrected so as to exclude all or some of the documents belonging to the selected cluster ( Step S3413). Thereafter, the process proceeds to step S3405, and thereafter, the processes of steps S3405 to S3413 are repeated.

  On the other hand, if there is no instruction to repeat the processing in step S3412 (No in step S3412), the entire series of processing is thereby terminated.

  As described above, according to the document classification apparatus according to the third embodiment, the vector correction unit 3301 can generate a new cluster that excludes the influence of a known cluster.

[Embodiment 4]
In the third embodiment described above, the configuration includes the vector storage unit and the vector correction unit. However, as in the fourth embodiment described below, the document expression space correction is performed instead of the vector correction unit. You may make it be the structure containing a part.

  The information processing system constituting the document classification device according to the fourth embodiment is the same as that of the first embodiment as shown in FIG. The hardware configurations of the server 101 and the client 102 are the same as those in the first embodiment as shown in FIGS.

  Next, a functional configuration of the document classification device according to the fourth embodiment will be described. FIG. 35 is a block diagram functionally showing the structure of the document classification device according to Embodiment 4 of the present invention. 35, the same components as those in FIG. 30 of the second embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  35, the document classification apparatus includes an input unit 3001, a language analysis unit 3002, a vector generation unit 3003, a classification unit 3004, a classification parameter instruction unit 3005, a classification result storage unit 3006, a cluster feature display unit 3007, a cluster feature. In addition to the calculation unit 3008, the classification system storage unit 3009, the cluster selection instruction unit 3010, the classification system browsing operation unit 3011, the vector storage unit 3501 and the document expression space correction unit 3502 are included.

  The vector storage unit 3501 stores the document feature vector generated by the vector generation unit 3003. Further, the document expression space correction unit 3502 calculates a document expression space for determining the similarity between the document feature vectors stored in the document feature vector storage unit 3501 from the subset selected by the cluster selection instruction unit 3010. It corrects based on the feature-value to do.

  Further, the classification unit 3004 classifies the document based on the similarity between the document feature vectors generated by the vector generation unit 3003 using the document expression space corrected by the document expression space correction unit 3502.

  It should be noted that the vector storage unit 3501 and the document expression space correction unit 3502 include the CPU 201 or 301 according to instructions written in a program recorded on a recording medium such as the ROM 202 or 302, the RAM 203 or 303, or the disk device 306 or the hard disk 316. Implements the function of each unit by executing instruction processing.

  Next, the contents of the document expression space correction unit 3502 will be described. In the vector correction unit 3302 in the third embodiment, the document feature vector is removed to eliminate the influence of the known cluster, but the multidimensional space itself representing the document feature vector is not changed. .

  Therefore, if it is desired to exclude the cluster formation feature selected by the operator as a result of the previous classification execution in the next classification execution, it is necessary to change the space representing the document vector.

  Therefore, a document expression space correction unit 3502 is provided to correct the document expression space. Here, as an example of changing the feature dimension of the document expression space, a description will be given of deleting a feature dimension having a high similarity to the center of gravity of the cluster selected by the operator.

  Since the centroid of the cluster selected by the operator can be expressed as a vector, by calculating the similarity between this cluster centroid vector and each feature dimension of the document expression space stored in the vector storage unit 3501, Discriminate feature dimensions with high similarity.

  Note that cosine, inner product, Euclidean distance, Mahalanobis distance, etc. are used as similar measures. Further, regarding the discrimination, discrimination by threshold processing that adopts a certain degree of similarity or higher as a deletion target, or discrimination by constant processing that adopts a certain number in descending order of similarity as deletion targets is used. Also, discriminant analysis or the like can be used.

  The document expression space correction unit 3502 calculates the feature dimension to be deleted as described above, and deletes the feature dimension. The feature dimension is deleted by deleting the row vector for the feature dimension determined from the “feature dimension (word)” × “document” matrix stored in the vector storage unit 3501. The document vector corrected by the document expression space correcting unit 3502 is stored in the vector storage unit 3501 for classification after the next time.

  Next, a series of processing procedures of the document classification device according to the fourth embodiment will be described. FIG. 36 is a flowchart showing a series of processing procedures of the document classification device according to the fourth embodiment. In the flowchart of FIG. 36, first, a document to be classified is input (step S3601).

  Next, the language of the input document is analyzed (step S3602), a document feature vector is generated based on the analyzed result, that is, the extracted word (step S3603), and the generated document feature vector is Stored (step S3604).

  After that, when there is an instruction for the classification parameter, and when there is an instruction for the classification parameter (Yes in step S3605), the document is classified according to the classification parameter instructed (step S3606). Information about the cluster is stored (step S3607).

  Next, the feature of the classified cluster is calculated (step S3608), and the calculated result is displayed (step S3609). It is determined whether or not a cluster has been selected from the displayed clusters (step S3610). If not selected (No in step S3610), the process proceeds to step S3605, and there is a classification parameter instruction again. Wait (step S3605).

  On the other hand, if a cluster is selected in step S3610 (Yes in step S3610), a classification system is generated and stored for the selected cluster (step 3611). At this time, information about the cluster input by the operator can also be stored. Thereafter, it is determined whether or not there is an instruction to perform repeated processing (step S3612).

  In step S3612, when there is an instruction to repeat the processing (Yes in step S3612), the row vector is deleted for the feature dimension determined from the matrix of “feature dimension (word)” × “document”. The document expression space is corrected (step S3613). Thereafter, the process proceeds to step S3605, and thereafter, the processes of steps S3605 to S3613 are repeated.

  On the other hand, if there is no instruction to repeat the processing in step S3612 (No in step S3612), the series of processing is thereby ended.

  As described above, according to the document classification apparatus according to the fourth embodiment, the cluster formation feature selected by the operator by the document expression space correction unit 3502 as a result of the previous classification execution is excluded at the next classification execution. And a new cluster can be generated in the excluded state.

[Embodiment 5]
In the third embodiment or the fourth embodiment described above, only one of the vector correction unit and the document expression space correction unit is included. However, as in the fifth embodiment described below, A configuration including both the vector correction unit and the document expression space correction unit may be adopted.

  The information processing system constituting the document classification apparatus according to the fifth embodiment is the same as that of the first embodiment as shown in FIG. The hardware configurations of the server 101 and the client 102 are the same as those in the first embodiment as shown in FIGS.

  Next, a functional configuration of the document classification device according to the fifth embodiment will be described. FIG. 37 is a block diagram functionally showing the structure of the document classification device according to the fifth embodiment of the present invention. In FIG. 37, the same components as those in FIG. 30 of the second embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  In the block diagram of FIG. 37, the document classification apparatus includes an input unit 3001, a language analysis unit 3002, a vector generation unit 3003, a classification unit 3004, a classification parameter instruction unit 3005, a classification result storage unit 3006, a cluster feature display unit 3007, a cluster feature. In addition to a calculation unit 3008, a classification system storage unit 3009, a cluster selection instruction unit 3010, a classification system browsing operation unit 3011, a vector storage unit 3701, a vector correction unit 3702, and a document expression space correction unit 3703 are included.

  The vector storage unit 3701 stores the document feature vector generated by the vector generation unit 3003. Further, the vector correction unit 3702 corrects the document feature vector stored in the document feature vector storage unit 3701 so as to be a document feature vector obtained by removing the document feature vector of the subset of documents generated by the classification unit 3004. To do.

  Further, the document expression space correction unit 3703 corrects the document expression space when judging the similarity between the document feature vectors stored in the vector storage unit 3701 based on the cluster feature selected by the cluster selection instruction unit 3010. To do.

  The classification unit 3004 classifies the document based on the similarity between the document feature vectors corrected by the vector correction unit 3702 using the document expression space corrected by the document expression space correction unit 3703.

  The vector storage unit 3701, the vector correction unit 3702, and the document expression space correction unit 3703 are in accordance with instructions written in a program recorded on a recording medium such as the ROM 202 or 302, the RAM 203 or 303, or the disk device 306 or the hard disk 316. Therefore, the function of each unit is realized by the CPU 201 or 301 executing instruction processing.

  Next, the contents of the vector correction unit 3702 and the document expression space correction unit 3703 will be described. In the fourth embodiment, the document belonging to the selected cluster is also used in the subsequent classification execution.

  In the fifth embodiment, by providing both the vector correction unit 3702 and the document expression space correction unit 3703, the document belonging to the selected cluster is removed at the next classification execution, and at the next classification execution Are not classified documents.

  In the fourth embodiment, the topic extraction aspect is emphasized, and it is assumed that a certain document is classified as a plurality of topics. For example, in an investigation related to networking, an “end user asks about software installation method”. Speaking of the answer, “Can't work as a system administrator because it comes”, this opinion can be categorized as the topic of “difficulty in understanding how to operate the software”, and “busy work of system administrators” There is a possibility to be classified by the topic.

  In any case, the fourth embodiment meets the need to extract both the cluster “difficulty in understanding how to operate software” and the cluster “busy work of system administrators”.

  On the other hand, since the topic extracted once is already known, there may be a case where the operator wants a different classification result as much as possible in the next classification. In the fifth embodiment, in order to respond to such a request, when the vector correction unit 3702 executes the classification of all or a part of the documents belonging to the cluster selected in the n-th classification after the next classification, It is to be removed from the object.

  Since the belonging document of the cluster for which the selection instruction is received by the cluster selection instruction unit 3010 is stored in the vector storage unit 3701 in the form of a column vector, the vector correction unit 3702 removes the inferior vector and executes the classification after the next time. A classification target document set is generated.

  Further, as in the fourth embodiment, the document expression space correction unit 3703 deletes the feature dimension from the matrix stored in the vector storage unit 3701 by the selected cluster.

  Next, a series of processing procedures of the document classification device according to the fifth embodiment will be described. FIG. 38 is a flowchart showing a series of processing steps of the document classification device according to the fifth embodiment. In the flowchart of FIG. 38, first, a document to be classified is input (step S3801).

  Next, the language of the input document is analyzed (step S3802), a document feature vector is generated based on the analyzed result, that is, the extracted word (step S3803), and the generated document feature vector is Stored (step S3804).

  Then, after waiting for a classification parameter instruction, if there is a classification parameter instruction (Yes in step S3805), the document is classified according to the classification parameter instructed (step S3806). Information about the cluster is stored (step S3807).

  Next, the feature of the classified cluster is calculated (step S3808), and the calculated result is displayed (step S3809). It is determined whether or not a cluster has been selected from the displayed clusters (step S3810). If not selected (No in step S3810), the process proceeds to step S3805, and the classification parameter is instructed again. Wait (step S3805).

  On the other hand, if a cluster is selected in step S3810 (Yes in step S3810), a classification system is generated and stored for the selected cluster (step 3811). At this time, information about the cluster input by the operator can also be stored. Thereafter, it is determined whether or not there is an instruction to perform repeated processing (step S3812).

  If it is determined in step S3812 that processing is to be repeated (Yes in step S3812), the document feature vector is corrected so as to exclude all or some of the documents belonging to the selected cluster (step S3812). Step S3813).

  Subsequent to step S3813, the document expression space is corrected by deleting the row vector for the feature dimension determined from the “feature dimension (word)” × “document” matrix (step S3814). Thereafter, the process proceeds to step S3805, and thereafter, steps S3805 to S3814 are repeated.

  On the other hand, if there is no instruction to repeat the processing in step S3812 (No in step S3812), the entire series of processing is thereby terminated.

  As described above, according to the document classification apparatus according to the fifth embodiment, the vector correction unit 3702 eliminates the influence of the known cluster, and the document expression space correction unit 3703 performs the previous classification execution. As a result, the formation characteristics of the cluster selected by the operator can be excluded at the next classification execution, and a new cluster can be generated in the excluded state.

[Embodiment 6]
In the second embodiment or the fourth embodiment described above, when repeated classification processing is performed, information on how many times a certain document has been selected is not considered, but the embodiment described below is used. As shown in FIG. 6, the selection information adding unit may be included, and the selection information may be displayed together with the cluster feature.

  The information processing system constituting the document classification apparatus according to the sixth embodiment is the same as that of the first embodiment as shown in FIG. The hardware configurations of the server 101 and the client 102 are the same as those in the first embodiment as shown in FIGS.

  Next, a functional configuration of the document classification device according to the sixth embodiment will be described. FIG. 39 is a block diagram functionally showing the structure of the document classification device according to the sixth embodiment of the present invention. In FIG. 39, the same components as those in FIG. 35 of the fourth embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  In the block diagram of FIG. 39, the document classification apparatus includes an input unit 3001, a language analysis unit 3002, a vector generation unit 3003, a classification unit 3004, a classification parameter instruction unit 3005, a classification result storage unit 3006, a cluster feature display unit 3007, and a cluster feature. In addition to the calculation unit 3008, the classification system storage unit 3009, the cluster selection instruction unit 3010, the classification system browsing operation unit 3011, the vector storage unit 3501, the document expression space correction unit 3502, a selection information addition unit 3901 is included.

  The selection information giving unit 3901 gives selection information indicating that the document is selected when all or some of the documents belonging to the document subset generated by the classification unit 3004 are selected. Further, the cluster feature display unit 3007 displays the cluster feature and the selection information given by the selection information giving unit 3901.

  It should be noted that the selection information adding unit 3901 is configured such that the CPU 201 or 301 executes instruction processing in accordance with an instruction described in a program recorded on a recording medium such as the ROM 202 or 302, the RAM 203 or 303, or the disk device 306 or the hard disk 316. To realize the function.

  Next, detailed contents of the selection information adding unit 3901 will be described. In the survey example, it is empirically known that unique opinions with high uniqueness are valuable. This is because it is often the opinion that the person in charge of planning the survey could not have anticipated.

  Therefore, when the documents belonging to the cluster selected by the operator are used for the subsequent classification execution, when each document is displayed on the cluster feature display unit 3007, how many times each document is selected. By indicating whether or not the document is used, it is possible to improve the distinguishability of a document that is used multiple times, and to improve the distinguishability of a document that has never been selected.

  FIG. 40 is an explanatory diagram showing a table 4000 provided in the classification result storage unit 3006 of the document classification device according to the sixth embodiment. In FIG. 40, a table is formed for each document ID, and the table 4000 records in which cycle each document is selected by the operator during classification. That is, “1” is recorded as selection information when selected, and “0” is recorded as selection information when not selected.

  For example, when the classification is executed four times, the document ID “1” indicates that the operator has selected at the first and second classification execution, and at the third and fourth classification execution. Indicates that it was not selected. On the other hand, the document ID “2” has never been selected, suggesting the possibility of an unknown opinion for the operator.

  Based on such information, when the cluster feature display unit 3007 displays the document to the operator, for example, the display may be changed according to the selected number of times. As the visual characteristics to be changed, for example, the density and saturation of characters and background colors can be considered.

  It is also possible to express the number of times selected directly by a number, a graph or the like. In any case, the display format is not limited to the above as long as the display format can visually identify a selected document and a document that has never been selected.

  Further, the selection information may be browsed by a browsing operation of the classification system browsing operation unit 3011.

  Next, the contents of processing of the selection information adding unit 3901 will be described. FIG. 41 is a flowchart illustrating a processing procedure of the selection information adding unit 3901 of the document classification device according to the sixth embodiment. In the flowchart of FIG. 41, first, classification processing is performed (step S4101), and subsequently, the first document is extracted (step S4102).

  It is determined whether or not the extracted document has been selected in the classification process in step S4101 (step S4103). If it is selected (Yes at step S4103), data “1” is recorded as selection information (step S4104). On the other hand, if not selected (No in step S4103), data “0” is recorded as selection information (step S4105).

  Next, it is determined whether or not processing has been completed for all documents (step S4106). If the processing has not been completed for all the documents (No at Step S4106), then the document is extracted (Step S4107), the process proceeds to Step S4103, and Steps S4103 to S4107 are repeated thereafter.

  On the other hand, in step S4106, when the processing is completed for all the documents (Yes in step S4106), the process proceeds to step S4101 and the classification process is performed again (step S4101). In this way, the processes in steps S4101 to S4107 are repeated as many times as the classification process is performed.

  As described above, according to the sixth embodiment, the selection information giving unit 3901 gives the selection information, and the cluster feature display unit 3007 displays the selection information. It is possible to improve the distinguishability of a document that has never been selected.

  The document classification methods described in the second to fifth embodiments are realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. This program is recorded on a computer-readable recording medium such as a hard disk, floppy disk, CD-ROM, MO, and DVD, and is executed by being read from the recording medium by the computer. The program can be distributed through the recording medium or as a transmission medium via a network such as the Internet.

  Next, information classification apparatuses according to Embodiments 7 to 16 will be described. In the embodiment of the present invention, when a collection of one or more sentences written in a natural language and the collection of one or more sentences is an object to be classified, it is referred to as a document. For example, published patent gazettes classified by IPC classification, etc., and newspaper articles classified into specific fields such as politics, economy, culture, science and technology, etc. are also documents, and claims and identification from them Even if one sentence is taken out, it is regarded as a document if it is a sentence included in a claim classification or a specific one sentence that can be classified according to usage. Embodiments 7 to 16 of the present invention will be described below in detail with reference to the drawings.

[Embodiment 7]
FIG. 42 is a block diagram showing the configuration of a document classification apparatus according to Embodiment 7 of the present invention. As shown in FIG. 42, the document classification apparatus according to the seventh embodiment includes a document input unit (document input unit) 5001 for inputting a document data group, and sets each document data based on a predetermined criterion. A document dividing unit (document dividing unit) 5002 that divides the document data into divided document data, and a document-divided document correspondence map generation unit (document-divided document correspondence map generation unit) 5003 that generates a map that associates the document data with the divided document data. I have.

  The document classification apparatus includes a divided document classification unit (divided document classification unit) 5004 that classifies divided document data, that is, divided documents, and a divided document classification result generation unit (divided document classification result) that generates divided document classification result information. Generating means) 5005, a document classification result generating unit (document classification result generating means) 5006 for generating classification result information of the document data using the document-divided document correspondence map and the divided document classification result information. Yes.

  The document dividing unit 5002, the document-divided document correspondence map generation unit 5003, the divided document classification unit 5004, the divided document classification result generation unit 5005, and the document classification result generation unit 5006 are stored in a shared or original program storage memory and program. Therefore, it has an operating CPU.

  Hereinafter, the document classification apparatus and the document classification method according to the seventh embodiment will be described in detail with reference to FIG. First, a document group is input by the document input unit 5001. The document input unit 5001 includes a keyboard, an OCR device, a removable recording medium, or network communication means, and inputs a document data group via any one of them.

  Then, the document dividing unit 5002 acquires the document data group, divides each document data based on a predetermined standard, and generates one or a plurality of divided document data from one document data. As a method for dividing the document data, the structure information of the document and the element information constituting the document are used, or the method specified by the user is used. However, the method is not limited here.

  FIG. 43 shows an example of generating a plurality of divided document data from document data performed by this document classification apparatus / document classification method. In the document 1 shown in this example, a plurality of news topics are described, and one day's topic is a document unit. As shown in the figure, since each news topic is separated by two line feed codes in this document, the document 1 which is one document is divided using this rule, and one is formed by one topic. Seven divided document data of documents 1-1 to 1-7 are generated. The document 1 before division can also be included as divided document data, but it is not included here.

  When the document is divided, the document-divided document correspondence map generation unit 5003 generates a map that associates the document data before division with the divided document data generated from the document data. For example, a map composed of an identifier uniquely indicating each document data and an identifier uniquely identifying each divided document data, or a map composed of an identifier uniquely identifying each divided document data is generated for each document data. . The method for associating the document data with the divided document data is not questioned here.

  FIG. 44 shows an example of generating a document-divided document correspondence map. In FIG. 44, documents 1 to 3 indicate document data, and divided documents 1 to 12 indicate divided document data. As shown in FIG. 44, each document data and divided document data is assigned an identification number (identifier) that can be uniquely identified, and the identification number of the document data and the identification number of the divided document data are shown in FIG. Corresponds in the table format shown in the lower left. In the case where arbitrary divided document data can be regarded as the same in the criteria used in document classification, their identification numbers may be the same.

  Subsequently, the divided document classification unit 5004 performs document classification for the divided documents. For example, language processing is performed on each divided document, the frequency of occurrence of each word included in the document is counted, and a feature vector that quantitatively represents the feature of each document is obtained based on the frequency. Document classification is performed by applying a chi-square method, a discriminant analysis method, or a cluster analysis method to these feature vectors.

  Next, as shown in FIG. 45, the divided document classification result generation unit 5005 generates divided document classification result information based on the divided document classification result.

  Here, the divided document classification result information is, for example, information related to the category to which each divided document data belongs (for example, “classification category” in the table “result of classifying divided document data into three categories” shown in FIG. 45). ”And“ Distance to representative value of affiliation category ”), information about individual affiliation categories generated (for example,“ representative value ”and“ information in the table of “category category information” shown in FIG. 45) Information in the section “number of belonging data (number of divided documents)”, information between generated belonging categories (for example, information in the table “distance between classification categories” shown in FIG. 45), and the like. The user can use various kinds of information as described above as basic data for classification result analysis.

  FIG. 45 is an example of generation of classification results when twelve divided document data are classified into three categories using their metric feature vectors. Metric three-dimensional vector of divided document data (The number of vector components is the number of all types of words that occur in the group of documents to be classified, but here, linear conversion into a three-dimensional vector in which some words are degenerated. For example, the Ward method, which is one of the cluster analysis methods, can be applied to the three categories.

  That is, each divided document data belongs to one of three categories as shown in the figure. The representative value of the affiliation category is the average value of the feature vectors of the affiliation divided document data (the center of gravity of the affiliation divided document data).

  The distance (corresponding to the similarity) with the representative value of the belonging category is, for example, for the divided document 3 in FIG. 45, the value of the divided document 3 in the section of the divided document data feature vector and the classification of the divided document 3 From the value of the term of the representative value of category 2, which is the category (the center of gravity of the assigned divided document data), it can be obtained from the following mathematical formula.

((3.00-2.66) ² + (2.00-2.00) ² ÷ (4.00-3.66) ² ) ^1/2 = 0.48
The smaller the distance from the representative value of the above-mentioned affiliation category, the higher the similarity with the average divided document belonging to that category.

  As the divided document classification result information, various statistics such as intra-category variance, inter-category variance, and the similarity range in each category can be generated in addition to those shown in FIG.

  Subsequently, the document classification result generation unit 5006 generates the document data classification result information as shown in FIG. 46, for example, using the document-divided document correspondence map and the divided document classification result information. In the example of FIG. 46, as shown, the divided document data to which each classification category belongs, its similarity (distance from the representative value of the belonging category), pre-division document data to which the divided document data belongs (affiliated document) , Document occupancy rate (ratio of divided document data belonging to the category), relative position (order) of the divided document data in the belonging document, ranking of similarity of the divided document data within the belonging category, etc. .

  In the above, the belonging document is obtained from the document-divided document correspondence map, and the other classification result information is obtained from the divided document classification result information. In addition to the information shown in FIG. 46, the document classification result generation unit 5006 has various statistics such as distribution within each category, deviation values within the category to which the divided document data belongs, contents of the document data and divided document data, and the like. Can also be used as classification result information.

  In the above description, all results are expressed in a table format with divided document data as a unit. However, classification results and document data can also be expressed in units. Moreover, the classification result information can be made not only textual representation but also graphical representation so that the user can easily understand.

  Thus, according to the present embodiment, one document is divided, the divided documents are classified, the correspondence between the pre-division document and the divided document is shown to the user, and the classification result of the divided document is displayed to the user. Therefore, if multiple topics or meanings are included in a single document, they are classified into categories that are limited to specific topics or meanings, or categories that differ from the categories intended by the user. Therefore, the user can understand the classification category well. Further, since the position of the divided document in the pre-division document (affiliation document) is also shown, the user can efficiently read the portion to be read in the document group.

[Embodiment 8]
FIG. 47 is a block diagram showing the configuration of the document classification apparatus according to Embodiment 8 of the present invention. As shown in the figure, the document classification apparatus according to the eighth embodiment adds a document storage unit (document storage unit) 5007 for storing document data and stores divided document data in addition to the configuration of the seventh embodiment shown in FIG. A document-divided document correspondence map storage unit (document-divided document correspondence map storage unit) that stores the document-divided document correspondence map generated by the divided document storage unit (divided document storage unit) 5008 and the document-divided document correspondence map generation unit 5003 Means) 5009 is provided. Each storage unit is constituted by a shared hard disk or a semiconductor memory, for example.

  With the above-described configuration, the document storage unit 5007 according to the present embodiment stores the information associated with the document data such as the contents of the document data and the document creator, creation date, and last modification date in an appropriate format. Further, when the document data has a metric feature vector composed of the elements together with the document contents, these are also stored. When the document input unit 5001 assigns an identifier that uniquely represents each piece of document data, this identifier can also be stored in an appropriate format.

  Further, the divided document storage unit 5008 stores the contents of the divided document data generated by the document dividing unit 5002 in an appropriate format, and also stores these in the case of having a metric feature vector. When an identifier that uniquely represents each of the divided document data is given, this identifier can also be stored in an appropriate format.

  Further, the document-divided document correspondence map storage unit 5009 stores the document-divided document correspondence map generated by the document-divided document correspondence map generation unit 5003 in an appropriate format.

  As described above, according to the eighth embodiment, the document data, the divided document data, and the document-divided document correspondence map are stored, so that the divided document data and the document-divided document correspondence map are not regenerated. For the same document data, classification results with different parameters such as the number of classifications, classification method, and various settings at the time of classification can be efficiently obtained. In addition, since the data necessary for classifying document data and generating the classification result is saved, the user can have a degree of freedom in time for the classification work, and the past document Reanalysis of the classification can also be performed at any time.

[Embodiment 9]
FIG. 48 is a block diagram showing the configuration of the document classification apparatus according to the ninth embodiment of the present invention. As shown in FIG. 48, the document classification apparatus according to the present embodiment has the divided document classification result information generated by the divided document classification result generation unit 5005 in addition to the configuration of the eighth embodiment shown in FIG. A divided document classification result storage unit (divided document classification result storage means) 5010 is provided. The divided document classification result storage unit 5010 is constituted by, for example, a shared hard disk or semiconductor memory.

  As described above, according to the third embodiment, the document data, the divided document data, the document-divided document correspondence map, and the divided document classification result information are stored. In addition to the effects of the eighth embodiment, Once classification is performed, the classification result can be expressed in various forms such as text expression, table expression, and graph expression. In addition, by storing the divided document classification result information, the user can have a degree of time freedom in the classification execution work and the classification result analysis work, and the past document classification result can be reproduced. Analysis can also be done at any time in various representation formats.

[Embodiment 10]
In Embodiment 10 of the present invention, in the document classification apparatus and document classification method of each of the above embodiments, as shown in FIG. 49, the document before division in a plurality of pieces of divided document data generated by the document division unit 5002 Document 1 that is data is included. Thereby, in this embodiment, the user can obtain not only the detailed document data classification structure obtained by classifying the divided document data but also the result of classifying the document data itself before the division. A classification structure in which macro classification structures are merged can be obtained.

[Embodiment 11]
In Embodiment 11 of the present invention, in the document classification apparatus and document classification method of each of the above embodiments, the document dividing unit 5002 divides document data based on the structure information of the document data. FIG. 50 shows an example of classification target document data or a document described in HTML format. Before dividing, the structure information is extracted from the document data in the HTML format as shown in FIG. 50, and the divided document data is generated from the document data by setting an appropriate dividing rule of the document using those structures. To do.

  That is, in this example, focusing on the tag <Ll> in the document data, the phrase “text having the tag <Ll> as one piece of divided document data” is a rule for generating the divided document data. By applying this rule to the document data, seven divided documents as shown in FIG. 50 are generated.

  As described above, even if the document does not have a specific structured document format such as HTML, XML, SGML, etc., a division rule is obtained from information on character size, character decoration, character color, font, etc. To generate a divided document. Further, when the document data is an image and is input by an OCR device or the like, it is possible to generate a division rule by using layout information of the original image and generate a divided document.

  Note that it is not necessary for all of the document data to be any divided document data. For example, in the example shown in FIG. 50, the character string “news topic (98/09/25)” is not adopted for the divided document.

As described above, according to the eleventh embodiment, by extracting structure information from document data and setting an appropriate document division rule using the structure information before dividing the document, it is possible to appropriately divide different topics. Therefore, it is possible to appropriately perform document classification in which the detailed classification structure of document data is known.

[Embodiment 12]
In Embodiment 12 of the present invention, as shown in FIG. 51, in the document classification apparatus and document classification method of Embodiments 7 to 10, a document element analysis unit (document (Element extracting means) 5011 and an element accompanying information extracting section (element accompanying information extracting means) 5012 for extracting element accompanying information such as parts of speech attached to elements extracted by the document element analyzing section 5011 (FIG. 51 is shown in FIG. 48). The document element extraction unit 5011 and the element associated information extraction 5012 are added to the illustrated embodiment 9), the document division unit 5002 is the element extracted by the document element analysis unit 5011, or The document data is divided using the element incidental information extracted by the element incidental information extraction unit 5012.

  As shown in FIG. 52, before performing document division, a document element analysis unit 5011 that is a natural language processing unit extracts those elements such as words from document data, and an element associated information extraction unit 5012 performs element associated information such as parts of speech. Is extracted, and an appropriate division rule for the document is set. Note that the document element analysis unit 5011 and the element associated information extraction unit 5012 are not newly provided, but the same means in the divided document classification unit 5004 can be used.

  In this embodiment, for example, as shown in FIG. 52, the document data is a collection of a plurality of news topics that do not have specific structural information, and each topic is a word “topic” + “number” + “new line”. In the case where it is described after the character string “code”, the structure as described above is recognized from the extraction results of the document element analysis unit 5011 and the element associated information extraction unit 5012, and “ The division rule “one character string of topic + number + line feed code and the character string enclosed by the character string or the document end symbol as one end” is generated.

  In more detail, first, only the noun and line feed code are extracted from the extracted word and its part-of-speech information, then the character string “topic + number + line feed code” and the document end symbol are detected, Remember their position at. Then, the division rule is applied to the document data to generate divided document data as shown in FIG.

  Note that it is not necessary to make all of the document data into any of the divided document data. For example, in the example shown in FIG. 52, the character string “News Topic (98/09/25)” is not adopted in the divided document. Further, in the above example, the case where an element and its accompanying information are extracted from the document data and the division rule is set has been described, but it is also possible to extract only the element and set the division rule from the element information. .

  Thus, according to the twelfth embodiment, by extracting the element information and the like from the document data and setting the document division rule using the extracted element information and the like, as in the eleventh embodiment, the document data It is possible to appropriately perform document classification that can understand the detailed classification structure.

[Embodiment 13]
In the thirteenth embodiment of the present invention, in the document classification device and the document classification method of the seventh to tenth embodiments, the document dividing unit 5002 divides the document data according to the designated range designated by the user. When the user designates the range of each divided document for the document data as shown in FIG. 53, the document dividing unit 5002 performs document division according to the specification.

  In the present embodiment, at the time of document division, the document division unit 5002 first displays an area designation object including left and right instruction points and area designation lines as an initial state on the screen at the top of the document. In this state, the user can select an area of each divided document by dragging one of the left and right instruction points using a pointing device such as a mouse and moving it up and down.

  At the time of designation, the document dividing unit 5002 changes the instruction pointer from black to white and the area designation line from a solid line to a broken line to indicate that the area selection process is being performed. In order to determine the selection area, it is only necessary to stop dragging the indication point at a desired position.

  Next, the user decides whether or not to make the selected area a divided document. If the area is not a divided area, the document dividing section 5002 displays the selected area in a shaded manner as shown in the drawing in order to explicitly display it.

  Thus, according to the present embodiment, the user can select each divided document data from the document data as desired, so that the detailed classification structure of the document data is known and suits the user's intention. Document classification can be performed.

[Embodiment 14]
In Embodiment 14 of the present invention, in the document classification device and document classification method of Embodiments 7 to 10, the document data is divided based on the number of characters in the document data, the number of sentences, or both the number of characters and the number of sentences. . For example, the document data shown in FIG. 54 is divided in units of approximately 200 characters.

  Here, approximately 200 characters are used as the unit, and even if it is an accurate 200 character unit, there is no guarantee that the end is a punctuation point, so the nearest punctuation point before and after the 200th character is the ending point of each divided document. Because. In this way, a divided document as shown in FIG. 54 is generated. Similarly, document division can be performed in units of a predetermined number of sentences, or document division can be performed based on both the number of characters and the number of sentences.

  As described above, according to the fourteenth embodiment, by dividing the document data based on the number of characters, the number of sentences, or both the number of characters and the number of sentences, contents of different topics are divided as different divided documents, Since there is a high possibility of classification, it is possible to perform document classification that shows the detailed classification structure of document data.

[Embodiment 15]
In the fifteenth embodiment of the present invention, in the document classification device and document classification method of each of the above embodiments, the document classification result generation unit 5006 uses the classification result information as information indicating the document data and the representative data attached to the document data. Present only information.

  For example, as shown in FIG. 55, a classification category name is displayed at the top, a keyword representing the category is displayed beside the category name, and divided document data belonging to the category is displayed as information indicating document data under the category name. For example, the document data name (document name) of the included document data is displayed. A document icon is displayed on the left side of each document data name, and when the document icon is designated, the contents of the document data are displayed.

  The document data names are arranged in the order (left side) of the document data names of the divided document data having a high similarity to the category representative value. When a plurality of divided document data generated from the same document data belong to the same classification category, only the document data name corresponding to the divided document data with the highest similarity is displayed. The keyword is a word having a high appearance frequency.

  As described above, according to the fifteenth embodiment, only the information indicating that the document classification result indicates the document data and the representative information accompanying the document data are displayed. Therefore, the user can obtain an overview of the detailed classification structure of the document data. It can be easily grasped.

[Embodiment 16]
In the sixteenth embodiment of the present invention, in addition to the document classification result presentation in the fifteenth embodiment, information indicating divided document data and information accompanying the divided document data are presented.

  For example, as shown in FIG. 56, a classification category name is displayed at the top, a keyword representing the category is displayed beside it, and divided document data belonging to the category is displayed as information indicating document data under the category name. For example, the document data name (document name) of the document data including “” is displayed.

  A document icon is displayed on the left side of each document data name, and when the document icon is designated, the contents of the document data are displayed. A divided document icon is displayed on the right side of the document data name. In the divided document icon, the position of the divided document data in the document data and the number of divided documents in the document data are displayed. Further, the divided document data in the document data can be displayed by designating the divided document icon.

  The document data names are arranged in order of the document data names of the divided document data having a high similarity to the category representative value. Further, when a plurality of divided document data generated from the same document data belong to the same classification category, the rank is displayed so that the rank of the similarity is understood.

  As described above, according to the sixteenth embodiment, the document classification result includes only the information indicating the document data and the representative information associated with the document data, and the information indicating the divided document data and the representative information associated with the divided document data. Since it is displayed, the user can easily understand which divided document is classified into the category due to the outline of the detailed classification structure of the document data.

  The document processing apparatus, document classification apparatus, document processing method, and document classification method of the present invention have been described above. However, the program for realizing the document processing method and the document classification method can be attached to and removed from the computer-readable recording medium. It is also possible to perform document processing and document classification according to the present invention in the information processing apparatus that has recorded and transferred the recording medium.

It is explanatory drawing which shows the hardware constitutions of the whole information processing system which comprises the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the hardware constitutions of the server in the information processing system which comprises the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the hardware constitutions of the client in the information processing system which comprises the document processing apparatus by Embodiment 1 of this invention. 1 is a block diagram functionally showing the configuration of a document processing apparatus according to Embodiment 1 of the present invention. FIG. It is explanatory drawing which shows the relationship between the item name and item value of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the data structure of the document memorize | stored in the document memory | storage part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another data structure of the document memorize | stored in the document memory | storage part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the list of the content of the extraction process performed by the feature extraction part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the list of the content of the process performed by the process part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the feature vector of each item of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the frequency | count of appearance for every word and its word ID of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the instruction | indication screen for the cross table preparation by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the cross table by which the result of the classification process by the output part of the document processing apparatus by Embodiment 1 of this invention was displayed. It is explanatory drawing which shows another cross table by which the result of the classification process by the output part of the document processing apparatus by Embodiment 1 of this invention was displayed. It is a block diagram which shows the detailed structure of the output part of the document processing apparatus by Embodiment 1 of this invention. It is a flowchart which shows the output procedure of the cross table of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is a block diagram which shows the detailed structure of the document memory | storage part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows another example of the screen display by the output part of the document processing apparatus by Embodiment 1 of this invention. It is a flowchart which shows a series of procedures of the document processing of the document processing apparatus by Embodiment 1 of this invention. It is a block diagram which shows functionally the structure of the document classification device by Embodiment 2 of this invention. It is explanatory drawing which shows an example of the display of the cluster characteristic display part of the document classification device by Embodiment 2 of this invention. It is a flowchart which shows the procedure of a series of processes of the document classification device by Embodiment 2 of this invention. It is a block diagram which shows functionally the structure of the document classification device by Embodiment 3 of this invention. It is a flowchart which shows the procedure of a series of processes of the document classification device by Embodiment 3 of this invention. It is a block diagram which shows functionally the structure of the document classification device by Embodiment 4 of this invention. It is a flowchart which shows the procedure of a series of processes of the document classification device by Embodiment 4 of this invention. It is a block diagram which shows functionally the structure of the document classification device by Embodiment 5 of this invention. It is a flowchart which shows the procedure of a series of processes of the document classification device by Embodiment 5 of this invention. It is a block diagram which shows functionally the structure of the document classification device by Embodiment 6 of this invention. It is explanatory drawing which shows the table provided in the classification result memory | storage part of the document classification device by Embodiment 6 of this invention. It is a flowchart which shows the procedure of the process of the selection information provision part of the document classification device by Embodiment 6 of this invention. It is a block diagram of the configuration of a document classification device showing Embodiment 7 of the present invention. It is explanatory drawing of the document classification device and document classification method by Embodiment 7 of this invention. It is another explanatory drawing of the document classification device and document classification method by Embodiment 7 of this invention. It is another explanatory drawing of the document classification device and document classification method by Embodiment 7 of this invention. It is another explanatory drawing of the document classification device and document classification method by Embodiment 7 of this invention. It is a block diagram of the configuration of a document classification device according to an eighth embodiment of the present invention. It is a block diagram of the configuration of a document classification device according to Embodiment 9 of the present invention. It is explanatory drawing of the document classification device and document classification method by Embodiment 10 of this invention. It is explanatory drawing of the document classification device and document classification method by Embodiment 11 of this invention. It is a block diagram of the configuration of a document classification device according to Embodiment 12 of the present invention. It is explanatory drawing of the document classification device and document classification method by Embodiment 12 of this invention. It is explanatory drawing of the document classification device and document classification method by Embodiment 13 of this invention. It is explanatory drawing of the document classification device and document classification method by Embodiment 14 of this invention. It is explanatory drawing of the document classification device and document classification method by Embodiment 15 of this invention. It is explanatory drawing of the document classification device and document classification method by Embodiment 16 of this invention.

Explanation of symbols

101 server 102 client 103 network 201 CPU
204 I / F
206 Disk device 301 CPU
306 Hard disk 308 Display 309 I / F
311 Keyboard 312 Mouse 313 Scanner 400 Bus 401 Input unit 402 Document storage unit 403 Selection unit 404 Feature extraction unit 405 Processing processing unit 406 Output unit 407 Graph drawing unit 408 Processing processing result holding unit 409 Analysis unit 410 Feature vector generation unit 1600 Instruction screen 1700 Cross table 1800 Mouse pointer 1801 Contents display screen 1901 Item value selection unit 1902 Totaling unit 1903 Table holding unit 2401 Bar graph display area 2501 Setting value storage unit 2502 Setting value transmission / reception unit 2503 Classification information storage unit 2604 Inquiry screen 2703 Classification information display screen 2802 Display area 3001 Input unit 3002 Language analysis unit 3003 Vector generation unit 3004 Classification unit 3005 Classification parameter instruction unit 3006 Classification result storage unit 3007 Class Feature display unit 3008 Cluster feature calculation unit 3009 Classification system storage unit 3010 Cluster selection instruction unit 3011 Classification system browsing operation unit 3110 Cursor 3301, 3501, 3701 Vector storage unit 3302, 3702 Vector correction unit 3502, 3703 Document expression space correction unit 3901 Selection Information adding unit 4000 Table 5001 Document input unit 5002 Document dividing unit 5003 Document-divided document correspondence map generating unit 5004 Division document classification unit 5005 Division document classification result generation unit 5006 Document classification result generation unit 5007 Document storage unit 5008 Division document storage unit 5009 Document-divided document correspondence map storage unit 5010 Division document classification result storage unit 5011 Document element analysis unit 5012 Element associated information extraction unit

Claims (4)

In a document classification device that classifies documents based on document contents,
An input means for inputting document data;
Language analysis means for analyzing the document data input by the input means to obtain language analysis information;
Based on the language analysis information obtained by the language analysis means , a document feature vector, which is column vector information representing the document data, is generated in a document expression space having a word as a feature dimension and the feature dimension as a row. Vector generating means for
Classification means for classifying documents based on the similarity between the document feature vectors generated by the vector generation means, and generating a subset of the documents;
Cluster feature calculation means for calculating a cluster feature that is a feature of a subset of documents generated by the classification means;
Cluster selection instruction means for selecting a desired subset from a subset of documents generated by the classification means;
Document feature vector storage means for storing the document feature vector generated by the vector generation means;
A vector correction unit for correcting the document feature vector stored by the document feature vector storage unit so that the document feature vector of the document belonging to the subset selected by the cluster selection instruction unit is removed;
From the document expression space used when judging the similarity between the document feature vectors stored in the document feature vector storage means ,
Corresponding to the feature amount expressed by the row vector of the feature dimension calculated from the subset selected by the cluster selection instruction means and the document included in the subset selected by the cluster selection instruction means A feature dimension row vector based on the document feature vector;
The feature dimensions of a high degree of similarity the document representation space to determine the,
A document expression space correcting means for correcting the line vector corresponding to the determined feature dimension by deleting it from the feature dimension of the document expression space;
With
The classification means uses the document expression space corrected by the document expression space correction means,
A document classification apparatus for classifying documents based on similarity between document feature vectors modified by the vector modification means. The document classification device further includes:
Display means for displaying the cluster feature calculated by the cluster feature calculation means;
Selection information giving means for giving selection information indicating selection when all or some of the documents belonging to the subset of documents generated by the classification means are selected;
The document classification apparatus according to claim 1, wherein the display unit displays the cluster feature and the selection information provided by the selection information addition unit. In a document classification method for classifying documents based on document contents,
Document classification device
An input process for inputting document data;
A language analysis step of obtaining language analysis information by analyzing the document data input in the input step;
Based on the language analysis information obtained by the language analysis means , a document feature vector, which is column vector information representing the document data, is generated in a document expression space having a word as a feature dimension and the feature dimension as a row. A vector generation step to
A classification step of classifying the document based on the similarity between the document feature vectors generated by the vector generation step, and generating a subset of the document;
A cluster feature calculation step of calculating a cluster feature that is a feature of a subset of documents generated by the classification step;
A cluster selection instruction step of selecting a desired subset from a subset of documents generated by the classification step;
A document feature vector storage step for storing the document feature vector generated by the vector generation step;
A vector correction step of correcting the document feature vector stored by the document feature vector storage step so that the document feature vector of the document belonging to the subset selected by the cluster selection instruction step is removed;
From the document expression space used when judging the similarity between the document feature vectors stored in the document feature vector storage step ,
Corresponding to a feature amount expressed by a row vector of a feature dimension calculated from the subset selected by the cluster selection instruction step and a document included in the subset selected by the cluster selection instruction step A feature dimension row vector based on the document feature vector;
The feature dimensions of a high degree of similarity the document representation space to determine the,
A document expression space correction step of correcting by deleting a row vector corresponding to the determined feature dimension from the feature dimension of the document expression space;
Including
The classification step uses the document expression space corrected by the document expression space correction step,
A document classification method, wherein documents are classified based on a similarity between document feature vectors modified by the vector modification step. The document classification method further includes:
A display step for displaying the cluster feature calculated by the cluster feature calculation step;
A selection information giving step for giving selection information indicating selection when all or some of the documents belonging to the subset of documents generated by the classification step are selected;
The document classification method according to claim 3 , wherein the display step displays the cluster feature and the selection information provided by the selection information addition step.

Images