JP2006202004A - Morpheme analysis device and its method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a morpheme analysis device and its method for efficiently and correctly analyzing a Japanese sentence including words (character string) constituted of alphabets including large characters. <P>SOLUTION: A word chain generating means 6 extracts words satisfying a part of speech chain which can be connected from an input text by using a word dictionary 3 and a grammar rule 4, and extracts words by re-retrieving the word dictionary 3 by converting large characters in the character string into small characters to prepare a word chain candidate column as for a character string constituted of alphabets including large characters, that is, an unknown character string whose notation is not registered in the word dictionary 3. A word selecting means 7 performs predetermined cost calculation for the word chain candidate string, and selects and outputs a word information string constituted of a series of word chain corresponding to the head to tail of the input text. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a morphological analysis technique that divides an input Japanese sentence (Japanese text) into words and outputs a word information string to which information such as readings and parts of speech is assigned for each word.

  For morphological analysis of Japanese sentences, a word dictionary and grammatical rules are used to extract (certify) words that satisfy a connectable part-of-speech chain from the input Japanese sentence, and the extracted words are information such as their readings and parts of speech. And generating a word chain candidate string that is a word chain candidate combined in correspondence from the beginning of the sentence to the end of the sentence (word chain generation processing), performing a predetermined cost calculation on the word chain candidate string, A method of selecting and outputting at least one word information string composed of a series of word chains corresponding from the beginning to the end of the Japanese sentence (word selection processing) is generally used (see Non-Patent Documents 1 and 2).

At this time, the extracted word is usually either a dictionary registered word registered in the word dictionary or an unknown word not registered in the word dictionary (when a part-of-speech chain cannot be generated by a dictionary registered word).
Kenji Yoshimura, et al. “Morphological analysis of written Japanese sentences using the minimum number of phrases”, Transactions of Information Processing Society of Japan, Vol. 24, no. 1, 1983, p. 40-46 Toru Hisamitsu, et al. “Proposal of morphological analysis by the minimum connection cost method and evaluation of computational complexity” IEICE technical report, Vol. 90, no. 116, 1990, p. 17-24

  As described above, in the conventional word chain generation process, a word (character string) whose notation is not registered in the word dictionary is treated as an unknown word (hereinafter, unknown word). Even if a word (English word) is registered in the word dictionary, the first character at the beginning of the sentence is capitalized (capitalized) or all characters are capitalized (capitalized) by emphasis etc. There is a problem that it is treated as an unknown word because it is different. For example, even if the word “restaurant” is registered in the word dictionary, it is unknown if “Restaurant” (first character capitalize) or “RESTAURANT” (all character capitalize) is not registered in the word dictionary. It will be treated as a word.

  This problem can be solved by registering the dictionary with `` Restaurant '' and `` RESTAURANT '' when registering the word `` restaurant '', but it causes a new problem that the size of the word dictionary increases accordingly, and `` When the word dictionary information related to “restaurant” is corrected, the word dictionary information must also be corrected for “Restaurant” and “RESTAURANT”, and the amount of work required to maintain (correct / update) the word dictionary accordingly. And the amount of processing increases, resulting in a new problem that leads to an increase in cost.

  In addition, if morpheme analysis is performed after lowering all alphabets in the input Japanese sentence, it is possible to deal with "Restaurant" and "RESTAURANT" mentioned above. It is difficult to distinguish words that are always capitalized words, such as or abbreviations, and that have the same spelling in lowercase letters from words that are registered in a separate dictionary (for example, “IT” is an abbreviation for “Information Technology”) "It" often represents "it", but "it" often represents "it"), and the input Japanese sentence is transformed to perform morphological analysis. If the notation is required, the original Japanese sentence must be saved and replaced with the original Japanese sentence after analysis, which requires a lot of memory and increases the amount of processing. To increase costs It creates a new problem of being connected.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a morphological analysis apparatus and method capable of efficiently and correctly analyzing a Japanese sentence including a word (character string) composed of alphabets including capital letters. To do.

  The present invention uses a word dictionary and a grammatical rule to generate a word chain supplementary sequence from an input Japanese sentence, perform cost calculation on the word chain candidate sequence, and output a word information sequence. When generating a word chain complement, for an unknown character string that is composed of alphabets including uppercase letters and whose notation is not registered in the word dictionary, uppercase letters in the character string are changed to lowercase. The word dictionary is re-searched above.

  According to the present invention, a Japanese sentence including capitalized English words can be analyzed efficiently and correctly.

  Hereinafter, the present invention will be described in detail with reference to the illustrated embodiments.

  FIG. 1 shows a morphological analyzer according to an embodiment of the present invention. In the figure, 1 is a first storage means, 2 is a second storage means, 3 is a word dictionary, 4 is a grammar rule, Central processing unit (CPU).

  The first storage means 1 is input directly from a keyboard (not shown) or the like, or read and input from a storage medium, or input from another device or the like via a communication medium (hereinafter referred to as input text). ) Is memorized.

  The second storage unit 2 stores a word chain complement sequence generated by a word chain generation unit described later.

  The word dictionary 3 is formed by registering a large number of words including at least one character together with information such as readings and parts of speech, and the grammatical rule 4 indicates whether connection is possible or impossible with respect to a chain of various parts of speech. These are actually stored and held in a storage device (not shown).

  The CPU 5 controls each unit described above according to the program shown in the flowcharts of FIGS.

  The word chain generation means 6 reads the input text from the first storage means 1 and extracts words that satisfy the connectable part-of-speech chain from the input text using the word dictionary 3 and the grammatical rule 4. For an unknown character string that is a character string consisting of alphabets including and whose notation is not registered in the word dictionary 3, extract the word by re-searching the word dictionary 3 with the capital letter in the character string lowercased Then, a word chain candidate string that is a complement of the word chain is generated by combining the extracted words together with information such as the reading and part of speech corresponding to the beginning of the input text to the end of the sentence, and the second storage means 2 Or a word satisfying a part-of-speech chain that can be connected from the input text using the word dictionary 3 and the grammatical rule 4 by reading the input text from the first storage means 1 Generating a word chain candidate string that is a candidate for a word chain that combines the extracted words together with information such as their readings and parts of speech in correspondence from the beginning of the input text to the end of the sentence, If the word dictionary 3 includes an unknown character string whose notation is not registered in the word dictionary 3, the word dictionary 3 is converted to the lower case after the capital letter in the character string is changed to lower case. Re-searching is performed, and when the corresponding word exists, the unknown alphabet character string is replaced with the word and stored in the second storage means 2.

  The word selection unit 7 reads the word chain complement sequence from the second storage unit 2 and performs a predetermined cost calculation on the word chain sequence (for example, any existing phrase minimum method, cost minimum method, etc. And the highest priority among a series of word chains corresponding from the beginning to the end of the input text is output as a word information string. At this time, a plurality of word information strings (ambiguity) may be output in the order of priority in the designated number.

  Hereinafter, the morpheme analysis method using the apparatus of the present invention will be described. First, the overall flow of morpheme analysis will be described with reference to FIG.

  First, when the input text is directly input from a keyboard or the like (not shown), or read and input from a storage medium or input from another device or the like via a communication medium, the CPU 5 stores the input text in the first storage unit 1. (S1).

  Next, the CPU 5 reads out the stored contents of the first storage means 1 by the word chain generation means 6 (s2), and uses the word dictionary 3 and the grammatical rule 4 to perform word chain generation processing to be described later to obtain the word A chain candidate sequence is created (s3) and stored in the second storage means 2 (s4).

  Finally, the CPU 5 reads the word chain supplementary sequence from the second storage unit 2 by the word selection unit 7 (s5), performs a word selection process similar to the conventional case, and outputs a word information sequence ( s6) The process is terminated.

  Next, the word chain generation processing by the word chain generation means 6 will be described in detail with reference to FIGS. As a predetermined dictionary search / connection check algorithm used in the following processing, any dictionary search / connection check algorithm (various existing technologies exist as shown in Non-Patent Documents 1 and 2) may be used. . The “dictionary search” is a search for the word dictionary 3, and the “connection check” is to check the connection possibility between words using the grammar rule 4 and the connection cost.

  FIG. 3 shows an example of a detailed flow of the word chain generation process in FIG.

  In step s11, an initial setting value in a predetermined dictionary search / connection check algorithm (the beginning of a sentence in an algorithm that processes from the beginning of a sentence, and the end of a sentence in an algorithm that processes from the end of a sentence) is set.

  In step s12, a word dictionary search is performed starting from the target character position using a predetermined dictionary search / connection check algorithm, and the process proceeds to step s13.

  In step s13, it is determined whether or not a matching (corresponding) word exists in the word dictionary 3. If the condition is satisfied, the process proceeds to step s19. Otherwise, the process proceeds to step s14.

  In step s14, it is determined whether or not the first character at the target character position is an alphabet. If the condition is satisfied, the process proceeds to step s15. Otherwise, the process proceeds to step s20. However, when the predetermined dictionary search / connection check algorithm is an algorithm for processing from the beginning of a sentence to the end of the sentence, the determination may be limited to whether or not the first character is an uppercase alphabet. In this case, it is only necessary to proceed to step s17 without performing the determination in step s16 described later.

  In step s15, continuous alphabets are extracted from the beginning of the target character string, and the process proceeds to step s16.

  In step s16, it is determined whether or not the extracted character string (alphabet character string) consisting of continuous alphabets includes uppercase letters. If the condition is satisfied, the process proceeds to step s17. Otherwise, the process proceeds to step s20.

  In step s17, the extracted alphabetic character string is converted to lower case letters, and a word dictionary search is performed using a predetermined dictionary search / connection check algorithm, and the process proceeds to step s18. Here, there are two types of lowercase letters: all lowercase letters (all lowercase letters), and the first letter (at the beginning of the sentence) with uppercase letters and all other lowercase letters (lowercase letters other than the first letter). Or the order of application may be determined in advance (for example, applied in the order of lowercase letters other than the first letter, then lowercase letters), and the first character string (in this example, lowercase letters other than the first letter) Only when there is no match, the next character string (in this example, all lowercase letters) may be searched, or only one of them may be searched.

  In step s18, it is determined whether or not a matching word exists in the word dictionary 3. If the condition is satisfied, the process proceeds to step s19. Otherwise, the process proceeds to step s20.

  In step s19, a word (dictionary search word) existing in the word dictionary 3 is added to the word chain supplementary string (if the lowercase alphabetical character string is matched, the notation is replaced with the original uppercase alphabetic character string). The target character position is updated in accordance with a predetermined dictionary search / connection algorithm, and the process proceeds to step s21.

  In step s20, the aforementioned alphabet character string is added as an unknown word to the word chain supplementary column, the target character position is updated according to a predetermined dictionary search / connection algorithm, and the process proceeds to step s21.

  In step s21, it is determined whether or not processing has been completed for all character strings. If the condition is satisfied, the process is terminated; otherwise, the process proceeds to step s12.

  In addition, in the dictionary search / connection check algorithm in which a word representing an unknown word (usually one character) is registered in the word dictionary 3 to be recognized as an unknown word, the matching word in steps s13 and s18 is a word. In the determination of whether or not it exists in the dictionary 3, if only the word representing the unknown word is searched from the word dictionary 3, it may be determined that “the condition is not satisfied”.

  FIG. 4 shows another example of the detailed flow of the word chain generation process in FIG.

  The processes in steps s11, s12, s13, s19, s20, and s21 are the same as those in the example of FIG. 3, and a word chain candidate string including unknown words similar to the case of the prior art is generated by these. .

  In step s31, it is determined whether or not there is an unknown character string (unknown word) composed of alphabets including capital letters that has not been processed in step s32 and subsequent steps in the word chain candidate strings generated in steps s11 to s21. judge. If the condition is satisfied, the process ends in step s32. Otherwise, the process ends.

  In step s32, the alphabet character string is converted to lower case and a word dictionary search is performed. This is the same processing as step s17. However, the connection check is performed on a word chained with the word in the already created word chain candidate string. After processing, the process proceeds to step s33.

  In step s33, it is determined whether or not a matching word exists in the word dictionary 3. If the condition is satisfied, the process proceeds to step s34. Otherwise, the process proceeds to step s31.

  In step s34, the unknown word in the word continuous candidate list is replaced with a word (dictionary search word) existing in the word dictionary 3. However, the notation remains the original alphabetic character string including upper case letters. After processing, the process proceeds to step s31.

Next, a specific processing example of morphological analysis according to the present invention will be described. In the following example, as input text,
It is assumed that the word “restaurant” and “Tokyo” are registered in the word dictionary 3, and the words “Restaurant” and “TOKYO” are not registered.

  The dictionary search / connection check algorithm uses an algorithm for processing from the sentence head to the sentence end. In the case of lowercase letters in step s17 in FIG. 3 and step s32 in FIG. 4, if the target alphabetic character string is all uppercase letters, the lowercase letters are applied in the order of lowercase letters and lowercase letters. In case of, only lower case letters are applied.

<Processing Example 1>
FIG. 5 shows an example of the case where the processing described in FIG. 3 is used as the word chain generation processing, and typical processing will be described below.

  First, in the process of step s11 by the word chain generation means 6, a sentence head is set as the target character position. In the process of step s12, a dictionary search is performed, ““ ”(part of speech: parentheses) matches, the process proceeds from step s13 to step s19, and“ “” (part of speech: parentheses) is added to the word chain candidate string. Shift the target character position one character backward.

  Next, the process proceeds from step s21 to step s12, a dictionary search is performed, and the process proceeds to step s13. Here, since there is no connectable word in the word dictionary 3, the process proceeds to step s14.

  In the process of step s14, since the head of the target character string is “R” and is an alphabet, the process proceeds to step s15.

  In the process of step s15, a continuous alphabet character string “Restaurant” is extracted from “R”, and the process proceeds to step s16.

  In the process of step s16, since “R” is a capital letter, the process proceeds to step s17.

  In the process of step s17, since “Restaurant” is capitalized only at the beginning, a dictionary search is performed for the character string “restaurant” in which all characters are converted to lowercase, and the process proceeds to step s18.

  In the process of step s18, “restaurant” (part of speech: noun) matches. Therefore, the process proceeds to step s19, where “Restaurant” (part of speech: noun) is added to the word chain complement, and the target character position is 10 characters. Shifting backward, the process proceeds from step s21 to step s12.

  In the process of step s12, a dictionary search is performed, and “” (part of speech: blank) exists as a matching word. Therefore, the process proceeds from step s13 to step s19, and “” (part of speech: blank) is replaced with the word chained complement. The target character position is shifted backward by one character.

  Next, the process proceeds from step s21 to step s12, a dictionary search is performed, and the process proceeds to step s13. Here, since there is no connectable word in the word dictionary 3, the process proceeds to step s14.

  In the process of step s14, since the target character string head is “T” and is an alphabetic character, the process proceeds to step s15.

  In the process of step s15, a continuous alphabet character string “TOKYO” is extracted from “T”, and the process proceeds to step s16.

  In the process of step s16, since “T” is a capital letter, the process proceeds to step s17.

  In the processing of step s17, since “TOKYO” is all capital letters, lower case letters are changed to the lower case, a dictionary search is performed for the character string “Tokyo”, and “Tokyo” (part of speech: noun) matches ( (“Tokyo” is not converted to lower case), the process proceeds from step s18 to step s19, “TOKYO” (part of speech: noun) is added to the word chain candidate string, the target character position is shifted backward, and step s21. To step s12.

  Since the subsequent processing is the same as the conventional dictionary search / connection check algorithm, description thereof is omitted.

  As a result, the word chain supplementary sequence 41 shown in FIG. 5 is generated.

  Next, in the process of step s6 by the word selection means 7, the word information sequence 42 shown in FIG. 5 is output by the word selection process similar to the conventional case.

<Processing example 2>
FIG. 6 shows an example of the case where the processing described in FIG. 4 is used as the word chain generation processing. Hereinafter, representative processing will be described.

  First, the word chain candidate string (after step s21) 51 shown in FIG. 6 is generated by the processing of steps s11 to s21 by the word chain generation means 6, and the process proceeds to step s31.

  In the process of step s31, since there is an unprocessed uppercase alphabet unknown word “Restaurant”, the process proceeds to step s32.

  In the process of step s32, since “Restaurant” is capitalized only at the beginning, a dictionary search is performed for the character string “restaurant” in which all characters are converted to lowercase, and the process proceeds to step s33.

  In the process of step s33, “restaurant” (part of speech: noun) exists and can be connected to the front ““ ”(part of speech: parentheses) and the rear“ ”(part of speech: blank). Move.

  In the process of step s34, the unknown word “Restaurant” is replaced with “restaurant” (part of speech: noun) searched for the dictionary, and only the notation is changed to “Restaurant”, and the process proceeds to step s31.

  In the process of step s31, since there is an unprocessed uppercase alphabet unknown word “TOKYO”, the process proceeds to step s32.

  In the process of step s32, since “TOKYO” is all capital letters, lowercase letters other than the head are converted, a dictionary search is performed for the character string “Tokyo”, and the process proceeds to step s33.

  In the process of step s33, “Tokyo” (part of speech: noun) exists and can be connected to the front “” (part of speech: blank) and the rear “” (part of speech: parentheses). Move.

  In the process of step s34, the unknown word “TOKYO” is replaced with “Tokyo” (part of speech: noun) searched for the dictionary, and only the notation is changed to “TOKYO”, and the process proceeds to step s31.

  In the process of step s31, since there is no unprocessed uppercase alphabet unknown word, the process ends.

  As a result, the word chain supplementary sequence (after the flow in FIG. 3) 52 shown in FIG. 6 is generated.

  Next, in the process of step s6 by the word selection means 7, the word information string 53 shown in FIG. 6 is output by the word selection process similar to the conventional case.

  Note that the description of the first and second storage means in the present invention is an expression based on a functional difference in what data is stored, and two separate storage devices are necessary in terms of hardware. It doesn't mean there is. In the embodiment, the example in which the word chain generation unit and the word selection unit are configured by a program on the central processing unit (CPU) has been described, but it goes without saying that each may be configured by hardware. .

The block block diagram which shows the morphological analyzer concerning embodiment of this invention The flowchart corresponding to the program of the morphological analyzer according to the embodiment of the present invention The flowchart which shows an example of the detailed content of the word chain production | generation process in FIG. The flowchart which shows the other example of the detailed content of the word chain production | generation process in FIG. Explanatory drawing which shows an example of the concrete morphological analysis process by this invention Explanatory drawing which shows the other example of the concrete morphological analysis process by this invention

Explanation of symbols

  1: first storage means, 2: second storage means, 3: word dictionary, 4: grammar rules, 5: central processing unit (CPU), 6: word chain generation means, 7: word selection means.

Claims (4)

In a morphological analyzer that divides an inputted Japanese sentence into words and outputs a word information string to which information such as reading and part of speech is given for each word,
First storage means for storing the input Japanese sentence;
The Japanese sentence is read from the first storage means, a word satisfying a connectable part-of-speech chain is extracted from the Japanese sentence using a word dictionary and grammatical rules, and at this time, a character string consisting of alphabets including capital letters For an unknown character string whose notation is not registered in the word dictionary, the word dictionary is searched again after lowering the capital letters in the character string, and the extracted word is read or part of speech. A word chain generation means for generating a word chain candidate string that is a word chain candidate combined in correspondence from the beginning to the end of the sentence along with information such as
Second storage means for storing the generated word chain candidate sequence;
A word information sequence consisting of a series of word chains corresponding to the sentence from the beginning to the end of the Japanese sentence, by reading a word chain complement from the second storage means, performing a predetermined cost calculation on the word chain complement A morpheme analyzer comprising: word selection means for selecting and outputting at least one of the words. Reading the Japanese sentence from the first storage means, using the word dictionary and grammatical rules, extracting words that satisfy the connectable part-of-speech chain from the Japanese sentence, and extracting the extracted words together with information such as their readings and parts of speech A word chain candidate string that is a word chain candidate combined in correspondence from the beginning to the end of the sentence of the Japanese sentence is generated, and is a character string composed of alphabets including capital letters in the word chain candidate string and is stored in the word dictionary If an unknown character string with no notation registered is included, the word dictionary is searched again after lowering the capital letters in the character string, and when the corresponding word exists, the unknown alphabet character string The morphological analysis device according to claim 1, further comprising a word chain generation unit that replaces the word with the word. In a morpheme analysis method for dividing an input Japanese sentence into words and outputting a word information string to which information such as reading and part of speech is added for each word,
Using a computer comprising a word dictionary, grammatical rules, first storage means, and second storage means,
The calculator
Storing the input Japanese sentence in the first storage means;
The Japanese sentence is read from the first storage means, a word satisfying a connectable part-of-speech chain is extracted from the Japanese sentence using a word dictionary and grammatical rules, and at this time, a character string consisting of alphabets including capital letters For an unknown character string whose notation is not registered in the word dictionary, the word dictionary is searched again after lowering the capital letters in the character string, and the extracted word is read or part of speech. Generating a word chain candidate string that is a word chain candidate combined in correspondence from the beginning to the end of the sentence along with information such as
Storing the generated word chain candidate string in a second storage means;
A word information sequence consisting of a series of word chains corresponding to the sentence from the beginning to the end of the Japanese sentence, by reading a word chain complement from the second storage means, performing a predetermined cost calculation on the word chain complement And a step of selecting and outputting at least one of the morpheme analysis methods. Reading the Japanese sentence from the first storage means, using the word dictionary and grammatical rules, extracting words that satisfy the connectable part-of-speech chain from the Japanese sentence, and extracting the extracted words together with information such as their readings and parts of speech A word chain candidate string that is a word chain candidate combined in correspondence from the beginning to the end of the sentence of the Japanese sentence is generated, and is a character string composed of alphabets including capital letters in the word chain candidate string and is stored in the word dictionary If an unknown character string with no notation registered is included, the word dictionary is searched again after lowering the capital letters in the character string, and when the corresponding word exists, the unknown alphabet character string The morphological analysis method according to claim 3, further comprising a step of replacing the word with the word.

Images