CN106250362A - Text segmentation device and text segmenting method - Google Patents

Abstract

The invention relates to a text segmentation device and a text segmentation method, which can effectively segment texts at appropriate positions. The computer searches for a first character string included in the text from character string division information that associates registered character strings divided into a plurality of words with the number of distinguished words (step 201 ). And, when the first character string corresponds to the registered character string, the computer divides the second character string of the first character string, which includes the words of the number of distinguished words corresponding to the registered character string, into the number of distinguished words. word (step 202).

Description

Text segmentation device and text segmentation method

technical field

The invention relates to a text segmentation device and a text segmentation method.

Background technique

In recent years, information on the Internet has increased dramatically, and businesses using big data have increased, so it is desired to efficiently process big data. In the case of a document that does not separate words and word representations with separator characters such as blanks, such as Japanese, Chinese, or Korean documents, morphological analysis is performed to calculate the frequency of occurrence of words.

Morpheme analysis is a process of dividing a text into morphemes and assigning part-of-speech information to each morpheme. The morphemes obtained by morpheme analysis are also sometimes handled as words. By performing such morphological analysis, it is possible to determine the relationship between words in the document and the part of speech of the word, and to divide the text in the document into words. However, since the processing load of morphological analysis is large, it takes a long time to process a large amount of text.

There is also known a word segmentation device that divides a sentence into two or more words at high speed (for example, refer to Patent Document 1). The word segmentation device selects from a word segmentation dictionary capable of storing one or more words and a word and a group of two or more segmented words as a result of segmenting the word, that is, one or more segmentation information. The pointer to the sentence at the head of the sentence acquires the word that matches the character string of the maximum length. Then, when there are two or more segmented words corresponding to the acquired word, the word segmenting device performs segmented word acquisition processing of acquiring two or more segmented words instead of matching words. After the word segmentation device moves the pointer of the sentence to the next character of the matching word, the segmented word acquisition process is performed until a word including the last character of the sentence is obtained, and the set of two or more words obtained by segmenting the sentence, that is, the first character is acquired. Split results.

A morphological analysis system capable of accurately and rapidly performing morphological analysis is also known (for example, refer to Patent Document 2). In the Chinese character string morpheme N-character registration dictionary of this morpheme analysis system, if the morpheme is two or more morphemes separated in the character string that becomes the morpheme if another arbitrary character string is followed and combined, then the same as The morpheme is associated and recorded together with the information of the separation position. The Chinese character string morpheme analysis program uses the longest agreement method to obtain the first morpheme candidate, and if the separation position information is recorded therein, it uses the longest agreement method to try to obtain the second morpheme candidate again from this position. In the hiragana morpheme conjugation list dictionary, a morpheme to which a plurality of hiragana morphemes are connected in consideration of the correctness of the grammatical connection is preliminarily recorded. The hiragana character string morpheme analysis program obtains the morpheme by comparing the hiragana morpheme connection list dictionary with character string data.

Patent Document 1: Japanese Patent Laid-Open No. 2014-106707

Patent Document 2: Japanese Patent Laid-Open No. 2002-32366

In the above-mentioned conventional word segmentation device or morphological analysis system, the segmentation position is determined based on information on only a part of the text, so the text is not always segmented at an appropriate position.

In addition, the above-mentioned problem is not limited to the case where the text is divided for counting the frequency of occurrence of words, but also occurs when the text is divided for other text analysis.

Contents of the invention

In one aspect, it is an object of the present invention to efficiently segment text where appropriate.

In one aspect, the text segmentation program causes a computer to execute the following processing.

(1) The computer retrieves the first character string included in the text from character string division information in which registered character strings to be divided into a plurality of words are associated with the number of differentiated words.

(2) When the first character string corresponds to the registered character string, the computer divides, among the first character string, a second character string comprising a word corresponding to the number of distinguished words corresponding to the registered character string into the number of distinguished words words.

According to the embodiment, text can be efficiently segmented at an appropriate position.

Description of drawings

FIG. 1 is a functional block diagram of a text segmentation device.

FIG. 2 is a flowchart of text segmentation processing.

FIG. 3 is a diagram showing character string division information.

FIG. 4 is a flowchart showing a specific example of text segmentation processing.

FIG. 5 is a functional block diagram of a text segmentation device that performs registration processing of the number of differentiated words.

FIG. 6 is a flowchart of the number-of-differentiated word count registration process.

FIG. 7 is a configuration diagram of an information processing device.

Text description of drawings

101...text division device; 111...storage unit; 112...segmentation unit; 121...character string division information; 501...divided word number determination unit; 701...CPU; ...Auxiliary storage device; 706...Medium drive device; 707...Network connection device; 708...Bus, 709...Removable recording medium.

detailed description

Hereinafter, embodiments will be described in detail with reference to the drawings.

For example, when the word segmentation device of Patent Document 1 is used to segment the text "そうはいってもっと向んでください", the text is segmented by the longest matching search of the dictionary for word segmentation. Therefore, although the correct segmentation result is "そう/はいって/もっと/向んで/ください", sometimes an undesirable segmentation result such as "そう/は/いっても/っと必んでください" is obtained.

This is considered to be due to the fact that although the division position may vary depending on the word immediately following a certain word, the division position is determined simply by the longest coincidence search without detecting a context wider than the word.

In addition, when using the morpheme analysis system of Patent Document 2 to segment the text of the compound word "Natural Language Processing Technology", the longest agreement search is performed again from the backward N character position, and when there is a second morpheme candidate word Next, use the split position.

Therefore, even when a correct segmentation result such as "natural language processing/technology" is obtained based on the first morpheme candidate, an incorrect segmentation result such as "natural/speech processing technology" may be adopted based on the second morpheme candidate . Likewise, the correct segmentation result of "Atomic Mechanics Society" is "atomic force/society", but sometimes an incorrect segmentation result of "atom/mechanics/society" is adopted based on the second morpheme candidate.

This is considered to be due to the fact that the context of the compound word is not detected, but the segmentation position is determined based on local information.

In this way, even if the segmentation position is determined based on information on only a part of the text, the context of the entire text is not detected, and thus an erroneous segmentation result may be generated. However, since sentences such as Japanese exist infinitely, segmentation results of all sentences are not registered in the dictionary.

FIG. 1 shows a functional configuration example of a text segmentation device according to an embodiment. The text segmentation device 101 in FIG. 1 includes a storage unit 111 and a segmentation unit 112 .

The storage unit 111 stores character string division information 121 in which registered character strings divided into a plurality of words are associated with the number of distinguished words. The segmentation unit 112 performs text segmentation processing with reference to the character string segmentation information 121 stored in the storage unit 111 .

FIG. 2 is a flowchart showing an example of text segmentation processing performed by the text segmentation device 101 in FIG. 1 . First, the segmentation unit 112 searches the first character string included in the text from the character string segmentation information 121 (step 201 ). Furthermore, when the first character string corresponds to the registered character string, the dividing unit 112 divides the second character string including the words of the number of distinguished words corresponding to the registered character string into the number of distinguished words in the first character string. words (step 202).

According to such a text segmentation device 101 , it is possible to efficiently segment a text at an appropriate position.

The text segmentation device 101 can be applied to text analysis for analyzing a large amount of text. For example, the text segmentation device 101 may be applied to statistical processing for calculating the frequency of occurrence of each word in the text.

FIG. 3 shows an example of character string segmentation information 121 for Japanese text. The character string segmentation information 121 in FIG. 3 is an n-gram table in which word-unit n-grams (n-grams) are registered as character strings, and corresponds to a dictionary used for text segmentation processing. Items of the n-gram table include item identification information (ID), n-gram, number of distinguished words, character string length, character type, and auxiliary words and positions of auxiliary verbs.

n-gram represents a character string including n words, and the number of distinguished words represents the number of words used as a segmentation result among the n words. The number of distinguishing words is an integer from 1 to n inclusive, but it is preferable to use a number of distinguishing words smaller than n. The string length indicates the number of characters included in the n-gram, and the character type indicates the type of characters in each word included in the n-gram. The character type "1" indicates that all characters included in the word are Hiragana or Katakana, and the character type "0" indicates that the word includes other characters. The positions of the auxiliary words and the auxiliary verbs indicate the occurrence positions of the auxiliary words and the auxiliary verbs included in the n-gram.

For example, the character string "そうはいっても" of ID "1" is a 4-gram composed of "そう", "は", "いって", and "も", the number of distinguishing words is 1, and the length of the string is 7. The character type "1111" indicates that each of the four words is hiragana or katakana, and the positions "2, 4" of the particle and the auxiliary verb indicate that the second and fourth words from the front end of the 4-gram are particles or auxiliary verbs.

The character string "そうはいはいと人" of ID "5" is a 4-gram composed of "そう", "はいはい", "と", and "人", the number of distinguishing words is 3, and the string length is 9. The character type "1110" indicates that the first to third words of the four words are hiragana or katakana, the fourth word includes other characters, and the position of particle and auxiliary verb "3" indicates that the third word is a particle or auxiliary verbs.

In addition, the position "-1" of the particle and the particle in the item such as ID "3" indicates that the n-gram does not include particle or particle.

Only the items of specific character strings of 2-gram to 4-gram are shown in FIG. 3 , but the n-gram table also includes items of other character strings of 2-gram to 4-gram not shown and 5- gram to 10-gram items. Also, n-grams in which n is 11 or more can be registered. By increasing n and registering a relatively long character string in the n-gram table, it is possible to detect a context wider than a word.

For example, an n-gram table can be automatically generated by analyzing the text of a plurality of files by high-precision morphological analysis processing. Even the same character string can be registered as different n-grams depending on the field of use. For example, the character string "atomic mechanics" can be registered as two 2-grams of "atomic mechanics" and "atomic mechanics". The method of determining the number of different words for each item will be described later.

FIG. 4 is a flowchart showing a specific example of text segmentation processing in FIG. 2 . First, the segmentation unit 112 uses the leading position of the text to be segmented as the start position, and searches for a character string starting from the start position in the text from the character strings registered in the character string segmentation information 121 by longest match search (step 401 ). Then, the division unit 112 checks whether the character string starting from the start position matches a character string in any item of the character string division information 121 (step 402 ).

When the character string starting from the start position does not match any of the character strings (step 402: No), the division unit 112 shifts the start position backward by one character (step 406), and repeats the processes after step 401 .

When the character string starting from the start position matches the character string of any one item (step 402: Yes), the division unit 112 refers to the number of distinguishing words of the item corresponding to the longest character string among the matching character strings (step 402: Yes). 403). Then, the dividing unit 112 divides the character string of the portion corresponding to the number of distinguished words from the start position in the text into the words registered in the item.

Next, the segmentation unit 112 shifts the start position backward by the number of characters of the character string corresponding to the number of words to be distinguished (step 404), and checks whether the end of the longest character string that matches is the end of the text (step 405) . When the end of the longest character string is not the end of the text (step 405 : NO), the dividing unit 112 repeats the processing from step 401 onwards.

When the end of the longest character string is the end of the text (step 405: Yes), the segmentation unit 112 divides the character string after the start position into the words registered in the item corresponding to the longest character string, and Finish processing.

For example, when the text to be segmented is "そうはいってもっとでんでください", the leading 2-character "そう" is used as the search object, and the ID is extracted by searching the n-gram table in FIG. Nine items of "1" to ID "9".

Items corresponding to the longest character string length "12" among these items are items of ID "3" and ID "4". However, both the character string "そうはいってもっとむこう" of the ID "3" and the character string "そうはいってずっとむこう" of the ID "4" do not match the text of the division object.

Items corresponding to the second longest character string length "9" are items of ID "5", ID "6", and ID "8". Among them, only the character string "そうはいってもっと" of the ID "8" matches the text to be divided, so the text to be divided is divided based on this item. In this case, the number of distinguished words of ID "8" is "1", and the division is performed at the position between the first "そう" and the second "はいって" among the three words included in the character string. The object's text, whose start position is shifted to the division position.

Next, if the remaining 2 characters "はい" at the front end of "はいってもっと必んでください" are searched for, and the n-gram table is searched by the previous coincidence search, ID "10" to ID "19" are extracted. 10 items.

Among these items, the item corresponding to the longest character string length "11" is the item of ID "11" and ID "12". However, both the character string "はいってもっとむこう" of the ID "11" and the character string "はいってずっとむこう" of the ID "12" do not agree with the remaining text.

Items corresponding to the second longest character string length "10" are items of ID "16" and ID "17". However, both the character string "はいってもっとむこう" of the ID "16" and the character string "はいってずっとむこう" of the ID "17" do not agree with the remaining text.

Items corresponding to the third longest character string length "8" are items of ID "14" and ID "19". However, both the character string "はいはいと秋册" of the ID "14" and the character string "はいってください" of the ID "19" do not match the remaining text.

Items corresponding to the fourth longest character string length "7" are items of ID "10", ID "13", and ID "18". However, since only the character string "はいってもっと" of the ID "18" matches the rest of the text, the rest of the text is divided based on this item. In this case, since the number of distinguishing words of ID "18" is "1", the position between the first "はいって" and the second "もっと" among the two words included in the character string Split the rest of the text, and shift the start position to the split position.

Next, if the remaining 2 characters "もっ" at the leading end of "もっと必んでください" are searched for, and the n-gram table is searched by the previous coincidence search, two IDs "20" and "21" are extracted. item.

The character string lengths of these items are all "6", but since only the character string "もっと必んで" of ID "20" matches the remaining text, the remaining text is divided based on this item. In this case, since the number of distinguishing words of the ID "20" is "1", the position between the first "もっと" and the second "入んで" among the two words included in the character string Split the remaining text, and shift the start position to the split position. Thereafter, the same division process is repeated for the remaining "入んでください".

According to such text segmentation processing, not all of the matched character strings are divided by the longest match search, but only a part thereof, and the remaining part can be included in the search target of the next longest match search. At this time, by registering a longer character string in the character string segmentation information 121, it is possible to gradually specify the segmentation result while comparing a plurality of registered character strings based on a wide range of contexts.

For example, it is also possible to determine the segmentation result only for the portion determined to be possible based on the context, and not to determine the segmentation result for the other portions, like the portion common to a plurality of registered character strings. In the example of "そうはいってもっと必んでください", the leading part of "そう" corresponds to the part determined to be possible, and the part after "はいって" corresponds to the other parts.

According to the text segmentation processing in FIG. 4 , a character string including unknown words that do not exist as words in the character string segmentation information 121 can also be segmented into a plurality of words.

For example, the text "XY Automobile Transportation Co., Ltd. の今日の支持は..." is the object of segmentation, and when "XY" is an unknown word, first, "X" at the front end is set as the start position. However, since the character string starting from "X" does not match any item of the character string division information 121, next, the start position is shifted by one character, and "Y" is set as the start position. However, since the character string starting from "Y" does not match any of the items of the character string division information 121, next, the start position is shifted by one character, and "from" is set as the start position.

Here, the 4-gram of "Automobile-Transportation-(Co., Ltd.)-の" is registered in the character string segmentation information 121, and if the number of distinguishing words is "3", the character string "Automobile-Transportation Co., Ltd." is divided Into "automobile", "traffic", and "(strain)" these three words. In addition, a character string "XY" preceding "Automobile Transportation Co., Ltd." is adopted as a word. Thereby, "XY Automobile Transportation Co., Ltd." can be divided into four words like "XY/Automobile/Transportation/(Co., Ltd.)".

In addition, a character string exists as a word in the character string division information 121, but even if a character string including the word is not registered, such an unregistered character string can be divided into a plurality of words.

For example, the text "そんなスリッパの..." is the object of segmentation. If "そんなスリッパ" is an unregistered character string, first, "そ" at the top is set as the start position. However, since the character string starting from "そ" does not match any item of the character string division information 121, next, the start position is shifted by one character, and "ん" is set as the start position.

However, since the character string starting from "ん" does not match any item of the character string division information 121, next, the start position is shifted by one character, and "na" is set as the start position. However, the character string starting from "な" does not match any item of the character string division information 121, so next, the start position is shifted by one character, and "ス" is set as the start position.

Here, the 2-gram "スリッパ-の" is registered in the character string segmentation information 121, and if the number of distinguishing words is "1", the character string "スリッパの" is divided into "スリッパ" and "の". words. And, the preceding character string "そんな" of "スリッパの" is adopted as a word. Thus, "そんなスリッパ" can be divided into two words like "そんな/スリッパ".

In this way, according to the text segmentation processing of FIG. 4 , even a character string not registered in the character string segmentation information 121 can be appropriately segmented. Therefore, it is not necessary to register a plurality of character strings including all the words in the character string segmentation information 121 , and it is only necessary to register only character strings that appear frequently statistically. Accordingly, it is possible to suppress an increase in the storage area for storing the character string division information 121 .

FIG. 5 shows an example of the functional configuration of a text segmentation device that performs registration processing of the number of differentiated words. The text segmentation device 101 in FIG. 5 has a configuration in which a segmented word number determination unit 501 is added to the text segmentation device 101 in FIG. 1 . The distinguishable word number determination unit 501 determines the distinguishable word count based on the attributes of the character strings in each item of the character string division information 121 , and registers the determined distinguishable word count in the character string division information 121 .

FIG. 6 is a flowchart showing an example of registration processing of the number of differentiated words performed by the number of differentiated word determination unit 501 . First, the distinguishable word number determination unit 501 takes the character string registered in one item of the character string segmentation information 121 as the processing object, extracts the attribute of the character string (step 601), and determines the character string corresponding to the processing object based on the extracted attribute. The number of distinguished words corresponding to the string (step 602).

Next, the distinguishable word number determination unit 501 checks whether or not there is a character string identical to the character string to be processed in other items (step 603 ). When the same character string exists (step 603: Yes), the distinguishing word count determination unit 501 changes the determined distinguishing word count to the number of words common to a plurality of identical character strings (step 604). Then, the distinguishing word number determination unit 501 registers the changed distinguishing word count in the item of the character string to be processed (step 605)

On the other hand, when the same character string does not exist (step 603: No), the distinguishing word count determination unit 501 registers the determined distinguishing word count in the item of the character string to be processed (step 605).

Next, the distinguishable word number determination unit 501 checks whether all the items of the character string division information 121 have been processed (step 606 ). When unprocessed items remain (step 606: No), the number-of-differentiated word determination unit 501 makes the character string registered in the next item a processing target, and repeats the processing from step 601 onward. Then, when all the terms have been processed (step 606: Yes), the number-of-differentiated word determination unit 501 ends the processing.

The distinguishing word number determination unit 501 may perform the distinguishing word count registration process in FIG. 6 before starting the text segmentation process in FIG. 4 , or may perform the distinguishable word count registration process in parallel with the text segmentation process.

The attributes of the character string to be processed extracted in step 601 of FIG. 6 can include the number of characters contained in part or all of the character string, the character types of the words contained in the character string, or the specified characters in the character string. At least one of the positions of the part of speech. The character type of a word represents, for example, hiragana, katakana, or other characters, and as predetermined parts of speech, for example, auxiliary words and auxiliary verbs are used.

Since a character string including words whose character type is hiragana or katakana often does not uniquely determine the division position, it is preferable to set such a character string so that the number of distinguishing words is smaller than that of a character string including words other than hiragana and katakana. The number of distinguishing words.

Also, since character strings including auxiliary words or auxiliary verbs often do not uniquely determine division positions, it is preferable to set the number of distinguishing words for such character strings to be smaller than the number of distinguishing words for character strings not including auxiliary words or auxiliary verbs.

In step 602 , the distinguishable word number determination unit 501 can determine the distinguishable word count z of the character string to be processed in the following procedure, for example.

First, when the n-th word (the last word) from the front end of the character string is a punctuation mark ("." or ","), the number-of-differentiating word determination unit 501 sets z=n, and When n words are not punctuation marks, z=n−1 is set.

Next, the number of distinguished words determination unit 501 checks the character type of the nth word and the part of speech of the (n-1)th word.

When the (n-1)th word is an auxiliary word or auxiliary verb, the character string up to the word corresponds to a text section (coherence of words), and between the (n-1)th word and the nth word There may be section boundaries. However, when the next n-th word is Hiragana or Katakana, there is not necessarily a boundary between the (n-1)th word and the n-th word. Conversely, when the nth word is a character other than hiragana or katakana, it can be said that there is a high possibility of a boundary between the (n−1)th word and the nth word.

Therefore, when the character type of the n-th word is "0" and the (n-1)-th word is an auxiliary word or an auxiliary verb, the distinguishing word number determination unit 501 does not change z.

On the other hand, when the character type of the n-th word is "1", or when the (n-1)-th word is a part of speech other than an auxiliary verb and an auxiliary verb, the number of distinguished words determination unit 501 performs the following procedure: make z decrease.

First, the number of distinguished words determination unit 501 checks whether or not k<z*3 is the number k of characters in the range from the beginning of the character string to the z-th word. In the case of z=n, k represents the total number of characters contained in the character string to be processed, and in the case of z=n-1, k represents the first to (n-1)th characters of the character string to be processed The number of characters contained in a word.

Since the division position is often not uniquely determined when the number of characters included in the character string is small, it is preferable to make z smaller. Therefore, in the case of k<z*3, the distinguishing word number determination unit 501 sets z=z−1.

Also, even when the number of characters included in the character string is large, when the character type is hiragana or katakana, the division position is often not uniquely determined. Therefore, when k≧z*3, and the character types of the first to (n−1)th words are all “1”, the number of distinguished words determination unit 501 also sets z=z−1.

In addition, the distinguishable word number determination unit 501 may compare k with other thresholds instead of comparing k with z*3, and may set z to a smaller value instead of z=z-1. .

In this way, by determining the number of distinguishing words based on the attributes of the character strings registered in the character string segmentation information 121, the portion for specifying the segmentation result is set according to the context of each character string. Thereby, the text can be segmented with high precision without reducing the processing speed.

In addition, in step 604, the distinguishing word number determination unit 501 compares character strings that are the same character string registered in a plurality of items and have different division positions from the front end, and sets the number of common words to for z. However, when the number of common words is equal to or greater than the number of distinguishing words determined in step 602 , the distinguishing word number determination unit 501 does not need to change the determined number of distinguishing words.

When registering the same character string with different division positions, by setting the number of words common to the division results of these character strings as the number of differentiated words, it is possible to reduce the possibility of dividing characters consisting of remaining words at wrong positions. string of risks.

For example, when "そうはいっても" of ID "1" in FIG. 3 is a character string to be processed, n=4, and the fourth word "も" is not a punctuation mark, so z=n-1= 3. Next, since the character type of the fourth word "も" is "1", it is checked whether k<z*3. In this case, since k=2+1+3=6, k<z*3=9, z=z-1=2 is determined (step 602).

Next, "そうはいっても" of ID "9" is the same character string, a word shared by "そう-は-いって-も" of ID "1" and "そうはいっても" of ID "9" Since it is only "そう", it is changed to z=1 (step 604).

When "そうはいってた" of ID "2" is a character string to be processed, z=2 is similarly determined (step 602). And, if it is assumed that the 3-gram "そう-はいって-た" is registered in another item, it is changed to z=1 (step 604).

In the case where "そうはいってもっとむこう" of ID "3" is a character string to be processed, since n=4, the fourth word "むこう" is not a punctuation mark, so z=n- 1=3. Next, since the character type of the fourth word "むこう" is "1", it is checked whether k<z*3. In this case, since k=2+4+3=9, k=z*3, but the characters of the first word "そう", the second word "はいって", and the third word "もっと" Since the types are all "1", it is determined that z=z-1=2 (step 602). Since the same character string as "そうはいってもっとむこう" does not exist in other items, it is determined that z=2.

When "そうはいはいと人" of ID "5" is a character string to be processed, n=4, and since the fourth word "人" is not a punctuation mark, it is set to z=n-1=3 . Next, since the character type of the fourth word "人" is "0" and the third word "と" is a particle, z=3 is determined (step 602). Since the same character string as "そうはいはいと人" does not exist in other items, it is determined that z=3.

The configuration of the text segmentation device 101 in FIGS. 1 and 5 is just an example, and some components may be omitted or changed according to the application and conditions of the text segmentation device 101 . For example, in the text segmentation device 101 of FIG. 5 , when the text segmentation processing is performed by an external device, the segmentation unit 112 can be omitted.

The flowcharts in FIG. 2 , FIG. 4 , and FIG. 6 are just examples, and part of the processing may be omitted or changed depending on the configuration and conditions of the text segmentation device 101 . For example, in step 401 of the text segmentation process in FIG. 4 , it is not necessarily necessary to perform the longest match search, and any one of the registered character strings matched by the previous match search may be used.

In steps 601 and 602 of the number-of-differentiation registration processing in FIG. 6 , the number-of-differentiation determination unit 501 may use, in addition to hiragana or katakana, types such as Chinese characters, English letters, numbers, symbols, etc., as included in the character string. The character kind of the word. In addition, the number of distinguished words determination unit 501 may use parts of speech such as nouns, verbs, adjectives, and adverbs in addition to auxiliary words and auxiliary verbs as predetermined parts of speech in the character string. The number of distinguishing words determining unit 501 may determine distinguishing words based on only one of the number of characters contained in a part or all of a character string, the character type of a word contained in a character string, or the position of a specified part of speech in a character string. number.

In the registration process of the number of distinguishing words in FIG. 6 , when the number of distinguishing words is not determined based on the attribute of the character string, the processing of steps 601 and 602 can be omitted. When the number of words common to a plurality of identical character strings is not registered as the number of differentiated words, the processing of steps 603 and 604 can be omitted.

The distinguishing word number determination unit 501 may register the distinguishing word count instructed by the user or the operator in the character string segmentation information 121 instead of performing the distinguishing word count registration process of FIG. 6 .

The character string segmentation information 121 in FIG. 3 is just an example, and other character string segmentation information 121 may be used depending on the configuration and conditions of the text segmentation device 101 . For example, when the text segmentation device 101 does not perform the processing of registering the number of distinguished words, the character string length, character type, particle, and particle position shown in FIG. 3 can be omitted. The character strings registered in the character string segmentation information 121 may not necessarily be in the form of n-grams, but may be in other forms indicating boundary positions between words. When dividing a text in a language other than Japanese, a character string in that language is registered in the character string division information 121 .

The text segmentation device 101 in FIGS. 1 and 5 can be realized using, for example, an information processing device (computer) as shown in FIG. 7 .

The information processing device of FIG. 7 includes a central processing unit (CPU) 701 , a memory 702 , an input device 703 , an output device 704 , an auxiliary storage device 705 , a medium drive device 706 , and a network connection device 707 . These constituent elements are connected to each other by a bus 708 .

The memory 702 is, for example, a semiconductor memory such as read only memory (ROM), random access memory (RAM), or flash memory. The memory 702 stores programs and data used for text segmentation processing or classification word number registration processing. The memory 702 can be used as the storage unit 111 in FIGS. 1 and 5 .

The CPU 701 (processor), for example, executes a program using the memory 702 to operate as the dividing unit 112 and the number of distinguished words determining unit 501 in FIGS. 1 and 5 .

The input device 703 is, for example, a keyboard, a pointing device, and the like, and is used for inputting instructions and information from a user or operator. The output device 704 is, for example, a display device, a printer, a speaker, etc., and is used for querying and outputting processing results to users or operators. The processing result may be a text segmentation result.

The auxiliary storage device 705 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a magnetic tape device, or the like. Secondary storage device 705 may be a hard drive or flash memory. The information processing device can store programs and data in the auxiliary storage device 705 in advance, and load them into the memory 702 for use. The auxiliary storage device 705 can be used as the storage unit 111 shown in FIGS. 1 and 5 .

The medium drive unit 706 drives a removable recording medium 709 and accesses its recorded content. The removable recording medium 709 is a storage device, a floppy disk, an optical disk, a magneto-optical disk, or the like. The removable recording medium 709 may also be a compact disc read only memory (CD-ROM), digital versatile disc (DVD), universal serial bus (USB) memory, or the like. A user or an operator can store programs and data in the removable recording medium 709 and load them into the memory 702 for use.

The computer-readable recording medium storing the program and data in this way is a physical (non-transitory) recording medium such as the memory 702 , the auxiliary storage device 705 , and the removable recording medium 709 .

The network connection device 707 is a communication interface connected to a communication network such as a local area network (LAN) or the Internet, and performs data conversion accompanying communication. The information processing device can receive programs and data from an external device via the network connection device 707 and load them into the memory 702 for use.

The information processing device can also receive instructions and information from the user terminal via the network connection device 707, perform text segmentation processing or differentiated word number registration processing, and transmit the processing result to the user terminal.

In addition, the information processing device does not need to include all the components in FIG. 7 , and some components can be omitted depending on the application and conditions. For example, the input device 703 may be omitted when no instruction or information input from the user or operator is performed, and the output device 704 may be omitted when no query to the user or operator or output of processing results is performed. In the case where the information processing device does not access the removable recording medium 709 or the communication network, the medium drive device 706 or the network connection device 707 may be omitted.

The disclosed embodiments and their advantages have been described in detail, but those skilled in the art can make various changes, additions, and omissions without departing from the scope of the present invention clearly described in the claims.

Regarding the embodiment described with reference to FIGS. 1 to 7 , the following additional notes are also disclosed.

(Note 1)

A text segmentation program that causes a computer to perform processing as follows:

Retrieving a first character string included in the text from the character string segmentation information corresponding to the registered character string to be divided into a plurality of words and the number of distinguished words; and

In the case where the first character string corresponds to the registered character string, the second character string of the first character string including the number of distinguishing words corresponding to the registered character string is divided into the distinguishing words The above distinguishes words by word count.

(Additional Note 2)

The text segmentation program described in Supplementary Note 1 is characterized in that the number of distinguished words is determined based on the attribute of the registered character string.

(Note 3)

The text segmentation program described in Supplementary Note 2 is characterized in that the attributes of the registered character string include the number of characters included in part or all of the registered character string, and the character types of the plurality of words included in the registered character string. , or at least one of the position of the prescribed part of speech in the registration character string.

(Note 4)

The text segmentation program described in any one of Supplements 1 to 3 is characterized in that the character string segmentation information includes a character string that is the same as the registered character string and is divided into For a character string of a plurality of words, the number of distinguished words is determined based on the number of words common to the registration character string and the character string divided at the different division positions.

(Note 5)

The text segmentation program described in any one of Supplements 1 to 4 is characterized in that the computer searches for the first character string from the character string segmentation information by a longest match search.

(Note 6)

A text segmentation device, characterized in that it includes: a storage unit that stores character string division information that associates a registered character string that is divided into a plurality of words with the number of distinguished words; Search for the first character string contained in the text in the string segmentation information, and if the first character string corresponds to the registered character string, include the above-mentioned character string corresponding to the registered character string in the first character string The second character string of words of the number of words is divided into the words of the number of words.

(Note 7)

The text segmentation device described in Supplementary Note 6 is characterized in that the number of distinguished words is determined based on the attribute of the registered character string.

(Note 8)

The text segmentation device described in Supplementary Note 7 is characterized in that the attributes of the registered character string include the number of characters included in part or all of the registered character string, the character types of the plurality of words included in the registered character string, , or at least one of the position of the prescribed part of speech in the registration character string.

(Note 9)

The text segmentation device described in any one of Supplements 6 to 8 is characterized in that the character string segmentation information includes a character string that is the same as the registered character string and is divided into For a character string of a plurality of words, the number of distinguished words is determined based on the number of words common to the registration character string and the character string divided at the different division positions.

(Additional Note 10)

The text segmentation device according to any one of Supplements 6 to 9 is characterized in that the segmentation unit searches for the first character string from the character string segmentation information by a longest match search.

(Additional Note 11)

A method for text segmentation, characterized in that,

The computer searches for a first character string included in the text from character string segmentation information associating registered character strings divided into a plurality of words with the number of distinguished words, and when the first character string corresponds to the registered character string and dividing a second character string of the first character string including words of the number of distinguished words associated with the registered character string into the words of the number of distinguished words.

(Additional Note 12)

The text segmentation method described in supplementary note 11 is characterized in that the number of distinguished words is determined based on the attribute of the registered character string.

(Additional Note 13)

The text segmentation method described in Supplementary Note 12 is characterized in that the attributes of the registered character string include the number of characters contained in part or all of the registered character string, and the character types of the plurality of words contained in the registered character string. , or at least one of the position of the prescribed part of speech in the registration character string.

(Additional Note 14)

The text segmentation method described in any one of Supplements 11 to 13 is characterized in that the character string segmentation information includes a character string that is the same as the registered character string and is segmented at a segmenting position different from the registered character string. For a character string divided into a plurality of words, the number of distinguishable words is determined based on the number of words common to the registration character string and the character string divided at the different division positions.

(Additional Note 15)

The text segmentation method described in any one of Supplements 11 to 14 is characterized in that the computer searches for the first character string from the character string segmentation information by a longest match search.

Claims (10)

1. A text segmentation device, characterized in that, possesses: a storage unit that stores character string division information corresponding to the registered character string to be divided into a plurality of words and the number of distinguished words; and A segmentation unit that searches for a first character string included in the text from the character string segmentation information, and when the first character string corresponds to the registered character string, divides the first character string that includes the above-mentioned The registered character string establishes a second character string of words corresponding to the above-mentioned number of differentiated words divided into the above-mentioned words of the above-mentioned number of differentiated words. 2. The text segmentation device according to claim 1, wherein: The number of distinguished words is determined based on the attribute of the registered character string. 3. text segmentation device according to claim 2, is characterized in that, The attributes of the above-mentioned registered character string include the number of characters contained in part or all of the above-mentioned registered character string, the character types of the above-mentioned multiple words contained in the above-mentioned registered character string, or the position of a specified part of speech in the above-mentioned registered character string at least one of the 4. The text segmentation device according to any one of claims 1 to 3, wherein: The character string division information includes a character string that is the same character string as the registered character string and is divided into a plurality of words at different division positions from the registered character string, and the number of differentiated words is based on the registered character string and the difference It is determined by the number of common words in the above-mentioned character strings that are divided at the division position. 5. text segmentation device according to claim 1, is characterized in that, The division unit searches for the first character string from the character string division information by longest match search. 6. A text segmentation method, characterized in that, The computer retrieves the first character string contained in the text from the character string segmentation information that associates the registered character string divided into a plurality of words with the number of distinguished words, And in the case where the above-mentioned first character string corresponds to the above-mentioned registration character string, the second character string of the above-mentioned first character string, including the words of the above-mentioned distinguishing word number corresponding to the above-mentioned registration character string, is divided into the above-mentioned distinguishing character strings Word count of the above words. 7. text segmentation method according to claim 6, is characterized in that, The number of distinguished words is determined based on the attribute of the registered character string. 8. text segmentation method according to claim 7, is characterized in that, The attributes of the above-mentioned registered character string include the number of characters contained in part or all of the above-mentioned registered character string, the character types of the above-mentioned multiple words contained in the above-mentioned registered character string, or the position of a specified part of speech in the above-mentioned registered character string at least one of the 9. The text segmentation method according to any one of claims 6 to 8, wherein, The character string division information includes a character string that is the same character string as the registered character string and is divided into a plurality of words at different division positions from the registered character string, and the number of distinguished words is based on the registered character string and the It is determined by the number of common words in the above-mentioned character strings that are divided into different division positions. 10. text segmentation method according to claim 6, is characterized in that, The computer searches for the first character string from the character string segmentation information by longest match search.