JP4071657B2 - Text processing device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates a representative representation of words by analyzing performed Allograph processing, the text processing apparatus for determining the frequency of occurrence of notation contained in a processed text.
[0002]
[Prior art]
Conventionally, techniques such as document search, document classification, and document analysis have been disclosed as techniques for processing a large amount of document information. Many of these document processing techniques use Japanese analysis processing for analyzing text in a document. The basis of Japanese analysis processing is morphological analysis that analyzes words. In many morpheme analysis methods, different notation processing is performed in which words having different notations are analyzed as the same word. As a result, words having the same meaning and different notations can be extracted as the same word, and accuracy deterioration due to different notations can be prevented in subsequent processing. Such different notation processing is performed by using a dictionary of different notation words (or synonyms) or by applying notation pattern rules. When a plurality of words having different notations are handled as the same word, when the extracted word information is presented to the user, a representative notation of the words is presented. Many use one type of representative notation.
[0003]
Patent Document 1 accepts a user input representing a concept of interest designated by a user, analyzes the electronically stored document, identifies a position where the concept of interest designated by the user is discussed, and electronically In order to provide the reader with a rough location of the part that discusses the concept of interest in the stored document, an indicator is displayed on the existing part to ensure that the desired information can be searched. Techniques to assist readers are disclosed.
[0004]
Patent Document 2 discloses a method and system for acquiring effective knowledge by extracting a concept having unique characteristics from a large amount of data including documents, a method for automatically finding a notable concept from a large amount of data, and A technique for providing a system and an analysis method and system excellent in usability in acquiring an effective characteristic concept from a large amount of data is disclosed.
[0005]
[Patent Document 1]
JP 2001-60206 A [Patent Document 2]
Japanese Patent Laid-Open No. 2001-75966
[Problems to be solved by the invention]
In the conventional technique, a predetermined notation such as dictionary information is used as a representative notation of the extracted word. However, in this case, even when the representative notation does not appear in the text to be analyzed or the appearance frequency is low, it is used as the notation of the extracted word, for example, the user who presented the word information has appeared. There was a problem that comprehension and comprehension were hampered by noting or noting that the appearance frequency was low.
[0007]
Recently, a concept expression method for expressing a concept included in a text by a combination of words included in the text has been devised. In such a concept expression method, when presenting the concept expression, if the notation of the word that is configured is different from the notation in the actual text, there is a high possibility that it will cause the user to understand and operate incorrectly.
[0008]
The present invention has been made in view of the above circumstances, and an object thereof is to determine the representative notation of a word analyzed by performing different notation processing based on the appearance frequency of the notation included in the text to be processed. .
[0009]
The present invention also provides a text processing device that uses morpheme analysis means for analyzing a plurality of words having different notations as one word when analyzing words included in the text. The purpose is to extract based on the appearance frequency of word notation in the text.
[0010]
The present invention is also a text processing apparatus that uses morpheme analysis means for analyzing a plurality of words having different notations as one word when analyzing a word included in the text, and a concept included in the text. In a text processing apparatus that uses a concept expression method that expresses a word using one or more words contained in text, the representative expression of the analyzed word is extracted based on the appearance frequency of the word expression in the text And
[0011]
The present invention is also a text processing apparatus that uses morpheme analysis means for analyzing a plurality of words having different notations as one word when analyzing a word included in the text, and a concept included in the text. In a text processing apparatus that uses a concept expression method that expresses a word using one or more words included in the text, the representative notation of words constituting the concept expression is the frequency of appearance of the word notation in the text portion that matches the concept expression It aims at extracting based on.
[0012]
[Means for Solving the Problems]
In order to achieve the above object, the invention according to claim 1 is a text processing device, wherein the text processing device performs dependency analysis and morphological analysis on each input text. A language analysis unit for obtaining an analysis result, a word extraction unit for extracting a word from each clause constituting the text based on the analysis result obtained by the language analysis unit, and an analysis obtained by the language analysis unit Based on the result, when each clause constituting the text includes an attached word having a specific expression, an intention expression extraction unit that extracts an intention expression that is an intention added to the clause, and the language analysis Analysis result obtained by the section, the word extracted by the word extraction section, phrase information consisting of the intention expression extracted by the intention expression extraction section, the word included in the phrase, the notation of the word, the appearance frequency of the notation Supported Generates a basic unit of a conceptual expression composed of only a word or a combination of a word and an intention expression for each phrase using the phrase information stored in the storage part and the phrase information stored in the storage part And a concept expression generation unit that generates a concept expression based on the relationship expression between the basic units of the concept expression based on the dependency relationship between clauses, and based on the words constituting the concept expression, the storage unit In the notation associated with the word, the notation having a high appearance frequency is extracted from the word list stored in (1), and the extracted notation is used as the notation of the words constituting the conceptual expression .
[0013]
According to a second aspect of the present invention, in the first aspect of the present invention, the appearance frequency is calculated by using the analysis result obtained by the language analysis unit to calculate the appearance frequency of each word in the text. It is characterized by that.
[0014]
According to a third aspect of the present invention, in the first aspect of the present invention, the word notation constituting the concept expression generated by the concept expression generation unit appears in the word notation in the text portion that matches the concept expression. It is characterized by using notation of frequently used words .
[0021]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
[0022]
<Example 1>
FIG. 1 is a diagram illustrating the configuration of an embodiment of the text processing apparatus according to the first embodiment. Text input unit 101 inputs text. If text is already recorded, it can be used as input. There may be one or a plurality of texts to be input, but the following description will be made assuming that a plurality of texts are input.
[0023]
The language analysis unit 102 performs language analysis on each input text. The apparatus according to the first embodiment uses morphological analysis as language analysis processing. The morpheme analysis to be used is a morpheme analysis process in which words having different notations are analyzed as one word. The process of analyzing different notation words as one word is performed by using a dictionary such as a different notation dictionary or synonym dictionary, or by generating different notation by applying a notation pattern rule for words in katakana notation. Can do. FIG. 2 is a diagram illustrating an example of different notation for language analysis according to the present embodiment.
[0024]
The word notation counting unit 103 generates a word list using the word information analyzed by the language analysis unit 102 and calculates appearance frequency information for each notation. An example of the word list is shown in FIG. Words are managed with IDs. Part of speech, appearance frequency, and notation information are managed as word information. As word notation information, the notation ID, notation, and notation frequency are managed for each different notation. In the case of a word that is useful, its end form is used as a notation.
[0025]
Thus, different notation words can be handled as one word by the word ID, and also the notation ID, notation, and notation frequency information for each different notation can be used.
[0026]
The language analysis result storage unit 104 stores and manages the word information of FIG. 3 generated by the word notation counting unit 103.
[0027]
When presenting the word information to the user, the notation information is referred to based on the word ID from the word information stored in the language analysis result storage unit. For example, the notation with the highest frequency by notation may be presented to the user as the representative notation of the word. A method of presenting a plurality of notations in descending order of the notation frequency is also conceivable.
[0028]
The text data processing apparatus according to the present embodiment is premised on using a concept expression method that expresses a concept included in a text using one or more words included in the text. First, the concept expression will be described.
[0029]
The concept expression used in this embodiment is based on clauses or inter-phrase relation information obtained as a result of language analysis of text. As the language analysis, for example, morphological analysis or phrase dependency analysis can be used. Morphological analysis analyzes words contained in text. Dependency analysis analyzes clauses included in text and analyzes clauses in a relationship between dependency and dependency as a relationship between clauses. In the case of the text “Software installation cannot be executed normally”, information as shown in FIG. 4 can be obtained as a result of language analysis.
[0030]
In FIG. 4, “self” represents an independent word, and “attached” represents an attached word. Autonomous words are part-of-speech words such as verbs, adjectives and nouns, and adjuncts are part-of-speech words such as particles and auxiliary verbs. A normal phrase consists of one independent word and zero or more attached words. Depending on the analysis method, there may be a result that a single phrase includes a plurality of independent words, but here, an analysis method for generating a phrase so that the phrase always includes only one independent word. Shall be used.
[0031]
The concept expression used in the present embodiment is expressed by a basic unit of the concept expression and a relation expression between the basic units. The basic unit of concept expression is expressed using tokens and intention expressions.
[0032]
A token is a word that expresses one meaning by itself, and an independent word can be used. For example, in FIG. 4, “software”, “installation”, “normal”, and “execution” are tokens. The token expression can use the token notation, or can be converted to a representative token notation.
[0033]
The intention expression is an expression representing the addition of meaning by an attached word in a phrase, and the intention added to the phrase is analyzed by extracting a specific expression pattern of the attached word. For example, the expression “~ not (auxiliary verb)” or “~ z (auxiliary verb)” means “cancellation”, the expression “can do (auxiliary verb)” means “possible”, “~ tai (auxiliary verb)” "Can be said to have added the meaning of" request "to the phrase. For example, intention expressions “possible” and “cancellation” are extracted from the phrase “cannot be executed” in FIG. The intention expression can be expressed as, for example, “(+ cancellation)” and “(+ possible−cancellation)”, where “+ XX” indicates that the intention expression is added, and “−XX” indicates that It means that no intention expression is added.
[0034]
As a basic unit of the concept expression, it is expressed by only a token, only an intention expression, or a combination of a token and an intention expression, for example, as shown in FIG.
[0035]
The combination of a token and an intention expression means that a specified token is included in a certain clause and the specified intention expression is added to the clause.
[0036]
The relationship between basic units indicates that there is a strong semantic relationship between basic units. A semantically strong relationship basically means being included in a clause having a dependency relationship. For example, if the relationship between basic units is represented by “⇒”, the conceptual expression “information⇒search” is “information” in the dependency clause and “search” in the dependency clause in the two clauses in the dependency relationship. ("Search for information"). By using the phrase dependency relationship as the relationship between basic units, instead of simply specifying the co-occurrence occurrence relationship in the text as in the word “software & installation”, which is a word logical expression often used in document search etc. , It can be specified that the basic unit appears in the text with a strong semantic relationship.
[0037]
In the clause dependency relationship, when a certain clause becomes a dependency clause, there is only one receiver clause, but a plurality of dependency clauses can relate to the same receiver clause (the clause 4 in the above example 1 is the clause 2). And is the receiving clause of clause 3). For this reason, there are two ways of expressing the relationship between basic units in the concept expression: when expressing the inter-phrase relationship of receiving clauses having a plurality of dependency clauses, and when not expressing them.
[0038]
In the case where the inter-phrase relationship of receiving clauses having a plurality of dependency clauses is not expressed, the conceptual representation is a simple one-dimensional list representation of basic units as shown in FIG.
[0039]
When expressing the inter-phrase relationship of receiving clauses having a plurality of dependency clauses, the concept representation is a tree representation of basic units as shown in FIG.
[0040]
When not expressing the inter-phrase relationship of receiving clauses with multiple dependency clauses, the concept expression is simple and easy for the user to understand, and it is easy to perform operations such as expanding the expression, but it is not possible to express complex clause dependency relationship structures There is. When expressing the inter-phrase relationship of receiving clauses having a plurality of dependency clauses, it is possible to express a complicated clause dependency relationship structure, but it is considered complicated and difficult for the user to perform. Although it can be used in both cases, in the following embodiments, it will be described as using an expression method in the case of not expressing the inter-phrase relationship of receiving clauses having a plurality of dependency clauses that are easy to understand and operate for the user.
[0041]
FIG. 8 shows an example of a conceptual expression that can be generated from a text in a case where the inter-phrase relationship of receiving clauses having a plurality of dependency clauses is not expressed.
[0042]
<Example 2>
FIG. 9 is a diagram illustrating a configuration of one embodiment of the text processing apparatus according to the second embodiment. The text input unit 201 inputs text. If text is already recorded, it can be used as input. There may be one or more texts to be input, but the following description will be made assuming that there is only one input text.
[0043]
The language analysis unit 202 performs language analysis of morphological analysis and dependency analysis on each input text. In morphological analysis, words included in text are analyzed. In dependency analysis, sentences and clauses included in the text are analyzed, and a clause having a relationship between dependency and dependency is analyzed as a relationship between clauses. For example, FIG. 4 shows an example of the analysis result when the sentence “Software installation cannot be executed normally” is analyzed. A word delimiter is represented by “/”. In addition, “self” above each word represents an independent word, and “attached” represents an attached word.
[0044]
A normal phrase contains one free word. Although there is a processing method for analyzing a single phrase so as to include a plurality of independent words, in this embodiment, it is assumed that a method for analyzing a phrase so as to always include only one independent word is used.
[0045]
The token extraction unit 203 extracts a token from each clause analyzed by the language analysis unit 202. From the word information in the phrase, a word that is an independent word part of speech is extracted and used as a token. FIG. 10 is extracted from the text example of FIG.
[0046]
The intention expression extraction unit 204 extracts an intention expression from each clause analyzed by the language analysis unit 202. A specific expression pattern is extracted from the word information in the phrase, and intention expression information is generated. For example, the intention expressions “cancellation”, “request”, “question”, and “possible” can be extracted when a word or expression pattern as shown in FIG. 11 is included. In the case of the text example of FIG. 4, “Sentence 4 intention expression: + possible + cancellation or less” is extracted as the intention expression.
[0047]
The text data structure storage unit 205 stores the text structure analyzed by the language analysis unit and the token and intention expression information extracted by the token extraction unit and the intention expression extraction unit.
[0048]
The text is stored in a data structure as shown in FIG. Each component holds the information shown in FIG. In this embodiment, as shown in FIG. 3, it is assumed that a word list in which a unique identifier is assigned to a word included in a text is generated and the word is managed. At that time, word notation information (notation ID, notation, frequency by notation) is also managed.
[0049]
FIG. 13 shows information held by each component. Each text is managed with a unique ID. Each text manages a sentence ID list included in the text, and each sentence manages its own sentence ID and a phrase list included in the sentence. The phrase manages its own phrase ID and the word ID-notation ID list, the related phrase ID list, and the received phrase ID included in the phrase. The word ID-notation ID is a combination of the word ID and the notation ID in the word list shown in FIG. As a result, not only the word but also which notation is managed. The related phrase ID list is an ID of a related phrase that receives the relevant phrase. As in the above example, since a plurality of clauses can be related clauses for one received clause, they are managed in the related clause ID list. The received phrase ID is an ID of a received phrase that is a related phrase. A dependency clause can take only one receiving clause. The clause also manages tokens and intention expressions extracted from the clause.
[0050]
It is also possible to hold the type of dependency relationship as information managed by the clause. For example, whether it is a combination modification or a combination modification. FIG. 14 shows an example of data held by a phrase.
[0051]
The concept expression generation unit 206 generates a concept expression from the phrase information stored in the text data structure storage unit 205. A concept representation basic unit is generated from clause tokens and intention expression information, and a concept representation is generated by a relationship expression between concept representation basic units based on the dependency relationship between clauses. From the text in the case of not expressing the inter-phrase relationship of receiving clauses having a plurality of dependency clauses, for example, it is possible to generate a conceptual expression shown when expressing the inter-phrase relationship of receiving clauses having a plurality of dependency clauses.
[0052]
In the present embodiment, the token notation of the concept expression basic unit constituting the concept expression is extracted based on the appearance frequency of the notation in the text.
[0053]
In the apparatus of the present embodiment, for example, when a conceptual expression “word ID1 → word ID3” is generated from the word ID in the word list of FIG. 3, the word notation having a high notation frequency in the text is used. In this example, the word ID3 has three types of notation (fast, fast, fast). For example, by using the notation with the highest notation frequency (fast), a concept such as “processing ⇒ fast” An expression notation can be generated. In addition, when there are a plurality of notations, a method may be considered in which a plurality of notations are written in descending order of the notation frequency. For example, “processing => (fast, fast, fast)”.
[0054]
In the apparatus according to the present embodiment, for example, when a conceptual expression of word ID1⇒word ID3 is generated based on the word ID in the word list of FIG. 3, the appearance frequency of the word expression in the text portion that matches the conceptual expression is used. Is used. The notation can be determined by the appearance frequency in the entire text, or the appearance frequency in the text portion to which the conceptual expression “word ID1 → word ID3” is applicable can be used. In other words, the notation is determined depending on which occurrence pattern of “processing ⇒ fast”, “processing ⇒ fast”, and “processing ⇒ fast” appears. For example, even if the notation “fast” appears most frequently in the entire text, if the notation “fast” is the most in the pattern “processing → word ID 3”, the concept expression “word ID 1 → word ID 3” The notation will be “processing ⇒ fast”.
[0055]
The concept expression storage unit 207 stores the concept expression generated by the concept expression generation unit 206.
[0056]
【The invention's effect】
As is apparent from the above description, according to the present invention, in analyzing a word included in a text, in a text processing apparatus using a morphological analysis unit that analyzes a plurality of words having different notations as one word, By extracting the representative notation of the analyzed word based on the appearance frequency of the word notation in the text, when the user presents or operates the extracted word or information using the extracted word, The word notation included can be presented.
[0057]
In addition, according to the present invention, when analyzing a word included in a text, the text processing apparatus uses a morphological analysis unit that analyzes a plurality of words having different notations as one word, and is included in the text. In a text processing apparatus that uses a concept expression method that expresses a concept to be expressed using one or more words included in the text, the representative expression of the analyzed word is extracted based on the appearance frequency of the word expression in the text Thus, when the concept expression information is presented to the user or operated, the word notation actually included in the text can be presented.
[0058]
In addition, according to the present invention, when analyzing a word included in a text, the text processing apparatus uses a morphological analysis unit that analyzes a plurality of words having different notations as one word, and is included in the text. In a text processing device that uses a concept expression method that expresses a concept to be expressed using one or more words included in the text, the representative notation of the words constituting the concept expression is represented by By extracting based on the appearance frequency, it is possible to present a word notation that appears in a text portion that actually matches the concept expression when the concept expression information is presented to the user or operated.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a configuration of one embodiment of a text processing apparatus according to a first embodiment.
FIG. 2 is a diagram illustrating an example of different notation in language analysis according to the first embodiment.
FIG. 3 is a diagram illustrating an example of a word list.
FIG. 4 is a diagram illustrating a result of language analysis in the case of a text “Software installation cannot be executed normally”.
FIG. 5 is a diagram illustrating an example of expressing a basic unit of concept expression.
FIG. 6 is a diagram illustrating a conceptual expression when a relation between clauses called receiving clauses having a plurality of dependency clauses is not expressed.
FIG. 7 is a diagram showing a conceptual expression when expressing an inter-phrase relationship called a receiving clause having a plurality of dependency clauses.
FIG. 8 shows an example of a concept expression that can be generated from text in a case where the inter-phrase relation of receiving clauses having a plurality of dependency clauses is not expressed.
FIG. 9 is a diagram illustrating a configuration of one embodiment of a text processing apparatus according to a second embodiment.
FIG. 10 is a diagram illustrating information extracted from the text example of FIG. 4 by the method of the second embodiment.
FIG. 11 is a diagram illustrating an example of intention expression information when extracting intention expressions of “cancellation”, “request”, “question”, and “possible”.
FIG. 12 is a diagram showing a data structure when text is stored.
FIG. 13 is a diagram illustrating information held by each component.
FIG. 14 is a diagram illustrating an example of data held in a phrase.
[Explanation of symbols]
101, 201 Text input unit 102, 202 Language analysis unit 103 Word notation counting unit 104 Language analysis result storage unit 203 Token extraction unit 204 Intention expression extraction unit 205 Text data structure storage unit 206 Concept expression generation unit 207 Concept expression storage unit

Claims (3)

A text processing device,
The text processing device is
A language analysis unit that obtains an analysis result by performing dependency analysis and morphological analysis on each input text;
Based on the analysis result obtained by the language analysis unit, a word extraction unit that extracts words from each clause constituting the text;
Based on the analysis result obtained by the language analysis unit, when each clause constituting the text includes an attached word having a specific expression, an intention expression that is an intention added to the clause is extracted. An intention expression extraction unit;
The analysis result obtained by the language analysis unit, the word extracted by the word extraction unit, the phrase information composed of the intention expression extracted by the intention expression extraction unit, the word included in the phrase, the notation of the word, the notation A storage unit for storing a word list associated with the appearance frequency of
Using the phrase information stored in the storage unit, for each phrase, a basic unit of concept expression composed of only a word or a combination of a word and an intention expression is generated, and the concept expression based on the dependency relationship between phrases A concept expression generation unit that generates a concept expression by a relation expression between basic units;
Based on the words constituting the concept expression, a notation having a high appearance frequency is extracted from the word list stored in the storage unit, and the extracted expression is used as the concept expression. A text processing apparatus characterized by being used as a notation of a constituent word. The frequency of appearance is
The text processing apparatus according to claim 1, wherein an appearance frequency for each notation of the word in the text is calculated using an analysis result obtained by the language analysis unit.   The notation of the word which has high appearance frequency is used for the notation of the word which comprises the concept expression produced | generated by the said concept expression production | generation part among the notation of the word in the text part applicable to a concept expression. The text processing device described.

Images