JPH05324707A - Natural language processor - Google Patents

Abstract

PURPOSE:To allow a user who has knowledge of an object field to automatically obtain more detailed information by inputting information on a modification relation, an equivalent, etc., and analyzing the information. CONSTITUTION:A sample sentence extracting device 3 extracts a sample sentence 5 from a target document 4. A sentence analyzing device 1 analyzes a sampled document of the field to be processed. The analysis result of the sentence analyzing device 1 is stored in an analytic storage device 2. When the sentence analyzing device 1 analyzes the document, the sample is not analyzed if the feature of the sentence to be analyzed is the same as that of a document which is analyzed before, so the information can effectively be gathered.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a natural language processing device, and more particularly to a dictionary creation support device in a machine translation device having a higher accuracy of the translation device.

[0002]

2. Description of the Related Art In a general-purpose natural language processing system, in order to deal with various language phenomena, it is usual to increase the amount of data such as dictionaries and grammars as much as possible, and to add proper usage information according to the case. However, when the processing target field or document is limited, the ambiguity is considerably eliminated, so
Not only is a great deal of use-specific information wasted, but there are many adverse effects such as incorrect processing being performed due to improper information addition.

In order to solve this point, for example, Japanese Patent Laid-Open No. 2-67679 proposes "translation processing system and device with dictionary creation support function". In this publication, statistical information such as the frequency of occurrence of words is extracted from the sample text written in the first language to be translated, and the user indicates the information to be extracted from the general-purpose dictionary based on the frequency information. Then, based on the instructed information, information suitable for the translation target text is extracted from the information in the general-purpose dictionary to create the user dictionary. However, the information that a user who has no knowledge about the system can directly register is limited to translated words and rough part-of-speech, and the more detailed information is required, the higher the ability of the user is required, and the incorrect registration. There are also many problems.

Sample sentences are extracted from the target document to be translated and subjected to linguistic analysis to create data (dictionary, grammar, etc.) customized for the target document. The more sample sentences there are, the more complete the data will be. However, considering cost performance, it is better to customize with as few sample sentences as possible. Therefore, I want to be able to complete the sample sentence with the already obtained information only once. Further, when word information (part of speech, translation, etc.) is obtained by a conventional sentence analysis device, the information may be ambiguous. In order to eliminate these ambiguities, statistical information is used to increase the analysis rate by using frequency and the like in the calculation of syntactic analysis. Something that is considered ambiguous from some sample does not need to get more samples,
Ambiguous words require more sample sentences because they require statistical data. Select more efficient sample sentences. Similarly, without requiring the user's advanced ability to provide information for disambiguating a word that has polysemy,
Automatic extraction is required.

[0005]

[Purpose] The present invention has been made in view of the above situation, and a user having knowledge of a target field inputs information such as dependency relations and translations, and analyzes the information. It is an object of the present invention to provide a natural language processing device capable of automatically obtaining detailed information.

[0006]

In order to achieve the above object, the present invention provides (1)
In a natural language processing device having a sentence analysis device that analyzes extracted sample sentences and an analysis result storage device that stores the analysis result of the sentence analysis device, when analyzing a document in a certain field to be processed by the sentence analysis device If the characteristics of the sentence to be analyzed are the same as those previously analyzed, it has a sample extracting means that does not analyze the sample, and (2) a sentence analyzer for analyzing the extracted sample sentence, In a natural language processing device having an analysis result storage device for storing the analysis result in the sentence analysis device, when analyzing a document in a certain field to be processed by the sentence analysis device,
Having a sample extracting means for selectively collecting a sentence including a word that causes polysemy when information for each word has polysemy, and (3) a document in a certain field to be processed in (1) or (2) above. When a sentence analysis device is used to analyze a word, a word that appears at the same time in a sentence that includes a word that causes polysemy when the word has polysemy is created as a dictionary of translated words of the polysemous word. 4) Above (3)
In the case of analyzing a document in a certain field to be processed by a sentence analysis device, a word syntactically related to a polysemous word among words included in a sentence including a polysemous word In a dictionary as the information of the translated word of the polysemous word, or (5) in a sentence analysis device that analyzes the extracted sample sentence and an analysis result storage device that stores the analysis result in the sentence analysis device, When analyzing a document in a certain field to be processed by a sentence analysis device, a sentence containing an ambiguous word is further selected by a sampling sentence extraction device when the meaning of each word is ambiguous, and the collected sample sentence is analyzed by the sentence analysis device. It is characterized by collecting information for disambiguation of an ambiguous word by analyzing. Hereinafter, description will be given based on examples of the present invention.

FIG. 1 is a block diagram for explaining an embodiment of a natural language processing apparatus according to the present invention. In the figure, 1 is a sentence analysis device, 2 is an analysis result storage device, 3 is a sample sentence extraction device, Reference numeral 4 is a target document, and 5 is a sample sentence. Figure 2
3 is a flowchart for explaining the operation of the natural language processing device according to the present invention. As an example, the case of analyzing an English sentence will be described below. First, the sentence analysis device 1 analyzes the following English sentences. This boy is wise. (1) Wise boy has good brain. (2) Good brain is heavy. (3) The sentence (1) is obtained as a sample sentence from the sample sentence extracting device 3 (step 1). If you divide this into words and search sentence analysis results by words, the analysis results will be empty at first (step
3) The sample sentence is displayed to the user, information for sentence analysis (dependency relation) is input by the sentence analysis device, and sentence analysis is performed based on this information (step 4). The input result of this user is for each sentence as shown in FIG. 3 (text information is added in the order of analysis for each sentence). From these, the sentence analysis device collects them in word units (step 5), and forms them into word units as shown in FIG.
The data is stored in the analysis result storage device 2 (step 6). Obtain sentence (2) as a sample sentence (step 1). Similar to sentence (1), it is divided into words, and it is checked whether the divided words have already been used in the analysis results so far.

To check whether or not it has already been used, the character string matching is performed by taking out one word from the word forming the sentence (2) and the line starting with "%" in the sentence analysis result storage device (step 1). To do. The character string of the sentence is stored in the line beginning with "%". At this point sentence (1)
Only the character string of the sentence of exists. The first word "wise" in sentence (2) is already used in sentence (1). The second word "boy" in sentence (2) is also used in sentence (1). Since the third word "has" of sentence (2) has not been used in sentence (1) yet, sentence (2) is adopted as a sample sentence and the result is analyzed by the sentence analysis device, and the result is shown in FIG. become that way. Similarly, sentence (3) is also analyzed. The word "heav" that composes sentence (3)
Since “y” does not have a character string in any “%” line of the sentence analysis results of sentences (1) and (2), it is adopted as a sample sentence and analyzed by the sentence analysis device. When trying to analyze, this boy is heavy. (4)

The words forming the sentence (4) are "this" and "b".
oy ”,“ is ”, and“ heavy ”are all the same as above (1) (2)
The feature is that it is included in the word group composed of the sentence group of (3). Therefore, sentence analysis has already been completed for each word, and information on each word has been obtained. Sentence analysis of sentence (4) is not performed (step 3). The fact that the characteristics are the same is that, for example, when a character string matching of the original sentence is performed and an English sentence with the characteristic that it is exactly the same as the already analyzed sentence comes, this sentence is not selected as a sample sentence. ..
Alternatively, when checking whether or not to use samples having the same characteristics as samples, it is possible to select the samples as having the same characteristics until a certain limit is exceeded, and to stop the selection when the limits are exceeded. For example, if the number of appearances of a word exceeds five, even a sample sentence consisting of the same word is adopted as a sample sentence, and if it consists of words exceeding five times, it is not adopted as a sample sentence. That's what it means. This will prevent the reliability of the obtained information analyzed from becoming too low due to too few samples.

In the end check (step 7), the number of sample sentences and the number of different words are limited in advance, and when the number exceeds the limit, the process ends. For example, the different words in the target document may be obtained in advance, and the number corresponding to 80% of the different words may be set as the number of different words with restrictions. Also,
It can also be limited by the degree to which the number of different words increases. The sentence may be a phrase, a chapter, a paragraph, a document, etc., as well as a text delimited by a period in English or a reading point in Japanese.

When a sample sentence is sentence-analyzed, the word "wi
Ambiguity may occur in the usage of "se". Ambiguity may occur only within a sample sentence and may be ambiguous compared to when other sample sentences were previously analyzed. For example, “wise” in sentence (1) means “wise”, but “wise” in the following sentence, The boy is wise in the brain. (5), means “being informed”.
In this way, when there are ambiguous samples for the same word, the priority of which meaning to store the meaning of "wise" becomes a problem. Therefore, when ambiguity occurs, a sample including the word in which ambiguity occurs is searched from the target document, and for example, He is wise in the brain. (6) He is wise in the machine. (7) Collecting and analyzing sentences, and obtaining a lot of "wise" information, we seek usability of each meaning. The meanings of “wise” in (6) and (7) above are “communication”. When the target field is narrowed down to a specific field, the information such as the meaning of a word is often decided almost uniquely,
In many cases, the meaning of a word, which is a simple parameter of frequency, may be determined.

When polysemy occurs as in the above example, as a method of obtaining information for eliminating the polysemy, a dictionary that simultaneously appears in a sentence including a word in which polysemy occurs is used as information of a translated word of the polysemous word. An example of creating will be described. In the case of ambiguity of “wise”, the word used at the same time as “wise” is 4 boy 1 this 1 when it is used in the translation of “wise” from the above sentence (1) (5) (6) (7). And, when it is "communication", it means machine 1 problem 1 brain 1 in 3 is 3 he 3 the 4 boy 1.

Therefore, the word form itself is memorized as "is", "boy", and "this" are used at the same time for the "wise" translation of the "wise" dictionary. In addition, the translated word for "communicate" is machin
Register e, problem, brain, to, is, he, the, this in the dictionary. It is sufficient to select a translated word that includes many words that appear in the sentence to be translated using this dictionary. Further, the frequency of appearance of words is also stored and used as the priority of words that appear at the same time, so that the calculation accuracy of the weight is improved.

Using the above example, words syntactically related to wise are more intelligent than the sentence analysis result in the “smart” sentence (1), “is”, and in the “sensible” sentence (5), “is” and “is”. In the “in” statement (6), it becomes “is” and in the “in” statement (7), it becomes “is” and “in”.

By thus registering syntactically related words as translated word information, it is possible to obtain characteristics of when and what meaning each ambiguous translated word is used for. Such information can be used for disambiguation as co-occurrence information at the time of translation. If the frequency information in the embodiment of claim 3 is used, the accuracy in disambiguation can be further improved. The related words are selected from the dependency relations (Figs. 5 (a) and 5 (b)) of sentence (1) and sentence (5) that directly relate to "wise" and those that "wise" directly receives. The range of related words can be preset to adjust the amount of related words.

The polysemy has not only the translated word but also the level of part of speech. The syntactically related words to eliminate these are stored together with the frequency as information for each meaning,
It is stored as information on the usage type of ambiguity words (singular or plural, time, etc.). When there is ambiguity at the part-of-speech level, remember which syntactically related words are used at the same time for nouns and verbs. By using the existing machine translation system, morphological analysis, or syntactic analysis to perform syntactic analysis in advance, and adopting the sentence that fails to be analyzed as a sample sentence, a sentence that includes information that is missing in the grammar or dictionary Can be effectively collected as a sample.

[0017]

[Effect] As is apparent from the above description, the present invention has the following effects. (1) In the sample sentence extracting device according to claim 1, since the sentence already analyzed as the sample sentence is not adopted as the sample sentence so that the sentence analysis is not performed anymore, the data of many different words can be efficiently and quickly obtained. You can collect sample sentences to obtain them. (2) In the sample sentence extracting device according to the second aspect, by collecting sentences including ambiguous words and increasing the number of samples of these sentences, a large amount of information can be obtained only for ambiguous words. As a result, the sample sentences can be collected efficiently. (3) In the dictionary creation support device according to the third aspect, when the translated word has ambiguous words, a dictionary is created for each translated word in which a word that appears at the same time as a word is stored. It is possible to store in the dictionary the features of which word is easily used at the same time as each word having a polysemous meaning. Conventionally, a dictionary was created by carefully considering which words were used by people with advanced technology at the same time as being cheap, but with this device, this was done automatically and statistically based on a large amount of information. The accuracy can be improved. (4) In claim 4, since only words syntactically related to a word having a polysemous meaning in a translated word are stored for each word having a polysemous meaning,
The memory requirement is smaller than that of the dictionary support device according to the third aspect. (5) In claim 5, not only the polysemy of the translation word but also the polysemy of the part-of-speech level is used to eliminate the polysemy, and the syntactically related words are used at the polysemy level or the cooccurrence description level at the same time. Ambiguity is resolved by memorizing in. It is possible to perform automatically without the need for advanced knowledge, as compared with the case of manually inputting the information for disambiguation.

[Brief description of drawings]

FIG. 1 is a configuration diagram for explaining an embodiment of a natural language processing device according to the present invention.

FIG. 2 is a flowchart for explaining the operation of the natural language processing device according to the present invention.

FIG. 3 is a diagram showing a sentence analysis result (sentence unit) of the present invention.

FIG. 4 is a diagram showing a result of sentence analysis of the present invention (a total result of word units).

FIG. 5 is a diagram showing a dependency relationship of a sentence according to the present invention.

[Explanation of symbols]

1 ... sentence analysis device, 2 ... analysis result storage device, 3 ... sample sentence extraction device, 4 ... target document, 5 ... sample sentence.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Miwa Shiraishi 1-3-6 Nakamagome, Ota-ku, Tokyo Stock company Ricoh Co., Ltd. (72) Masumi Narita 1-3-6 Nakamagome, Ota-ku, Tokyo Shares Within Ricoh Company (72) Inventor Yoshihisa Oguro 1-3-6 Nakamagome, Ota-ku, Tokyo Within Ricoh Company, Ltd.

Claims (5)

[Claims] 1. A natural language processing device having a sentence analysis device for analyzing an extracted sample sentence and an analysis result storage device for storing an analysis result in the sentence analysis device, wherein a document in a certain field to be processed is sentence analyzed. A natural language processor characterized in that it has a sample extraction means that does not analyze the sample, when analyzed by the device, if the characteristics of the sentence to be analyzed are already the same as previously analyzed. 2. A natural language processing device having a sentence analysis device for analyzing an extracted sample sentence and an analysis result storage device for storing an analysis result of the sentence analysis device, wherein a document in a certain processing target field is subjected to sentence analysis. A natural language processing apparatus comprising sample extracting means for further selectively collecting a sentence including a word that causes ambiguity when information for each word causes ambiguity when analyzed by the apparatus. 3. When analyzing a document in a certain field to be processed by a sentence analysis device, when there is ambiguity in a translated word of a certain word, words that appear at the same time in a sentence including a word in which the ambiguity occurs are information of the translated word of the ambiguous word. The natural language processing apparatus according to claim 1 or 2, wherein a dictionary is created as. 4. When a document of a certain processing target field is analyzed by a sentence analysis device, when a polysemous word occurs in a translated word of a certain word, a polysemous word among the words included in the sentence including the polysemous word is syntactically defined. 4. The natural language processing apparatus according to claim 3, wherein a dictionary is created using related words as information about translated words of the polysemous words. 5. A sentence analysis device for analyzing an extracted sample sentence, and an analysis result storage device for storing an analysis result of the sentence analysis device, wherein when a document in a certain field to be analyzed is analyzed by the sentence analysis device, When a polysemous information occurs in each information, a sentence including a polysemous word is further selected by a sampling sentence extraction device, the collected sample sentence is analyzed by a sentence analysis device, and information for disambiguation of a polysemous word is obtained. A natural language processing device characterized by collecting.