JP2002245061A - Keyword extraction - Google Patents

Abstract

(57) [Summary] [Problem] It has been difficult to construct a database for easily and accurately searching a large amount of text data such as web pages. SOLUTION: Data of a web page, which is a set of text data, is collected by a traveling engine, and the collected data is subjected to morphological analysis to extract words. For these words, TFIDF, which is a biased frequency of appearance, is calculated, and only predetermined words are extracted as keywords. Using these words, a vector representing the text data is calculated to construct a database. At the time of a search, a search sentence is input, a keyword is cut out from the sentence, a vector represented by the keyword is compared with a database, and similar sites are output. Rather than a simple word comparison, similarity in a vector represented by a word characterizing a document can be determined, and search accuracy can be improved.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for searching and classifying text data having a certain unity. More specifically, the present invention relates to a technique for efficiently extracting keywords and assigning classifications to construct a database. , Generate a summary sentence,
Alternatively, it relates to a technique for performing a search.

[0002]

2. Description of the Related Art Conventionally, various methods have been proposed to handle a large amount of text data, such as a web page accessible on the Internet. For example, in a search engine that exists on a network such as the Internet for the purpose of searching for such web pages, when a client inputs a search keyword to the search engine, a text including the corresponding keyword exists. The page can be referenced.

[0003] In such a search, before executing a search by a client, the user goes around each site and collects all text data existing in the site to construct a database, or constructs a database from the text of the top page. This is done by a reserved method. In this case, the extraction of words from the text data is performed by preparing a thesaurus or the like in advance and extracting only words existing in the thesaurus, or simply extracting a series of kanji and katakana as words. .

In order to identify relevant text data by search, there is a method of determining only whether or not a keyword exists. However, a vector of a large number of words extracted from the text data is calculated and calculated from the keyword. A method for determining the degree of similarity with a vector to be performed has also been proposed. This means that if there are 10,000 words in the thesaurus, a space consisting of these 10,000 words is assumed, and what kind of vectors words contained in specific text data form in this space is calculated. Keep it. In this case, each sentence of the vector is changed according to the frequency of appearance of the word. For example, as shown in FIG. 18A, in a certain text data A, the word “mountain” is three times as a keyword,
Assuming that the word "river" appears five times, the vector A of this text data is expressed as a vector having "mountain" and "river" as components as shown in FIG. You. Similarly, text data B includes
If "mountain" appears only twice and "river" does not appear,
The vector B becomes a vector overlapping the “mountain” axis,
Assuming that “river” appears only three times in the text data C, the vector C becomes
This is a vector that overlaps the “river” axis. On the contrary,
When the keyword “mountain, river” is given, the vector D becomes a unit vector having “mountain” and “river” as shown in FIG. Is determined to have the highest similarity.

[0005]

However, there is a problem that a large amount of text data cannot be efficiently handled by such a search technique. That is, in a simple keyword search, there is too much noise, and the searched text data becomes enormous. Taking an Internet site as an example, text data of sites all over the world connected to the Internet is acquired by a cyclic engine, and words included in these are registered as keywords, for example, "PC" And search for hundreds of thousands of sites. This is because even if a part of the text data states that "the questionnaire can be accessed also from a personal computer", this is true.

[0006] On the other hand, in a method in which a vector of the entire data is obtained by using words included in the text data, a similarity is determined using this vector, and the result is reflected in a search result. Since the number of words included is large, there is a problem that a great deal of time and labor are required for the calculation.

[0007] Such a problem has been a problem not only in the search but also in a technique for handling text data for a natural language (text), such as construction of a search database and creation of a summary sentence.

[0008] The device of the present invention solves these problems,
An object of the present invention is to propose a calculation amount and realize a technology for handling text data with high accuracy.

[0009]

Means for Solving the Problems and Action / Effect Thereof A keyword extracting method according to the present invention for solving at least a part of the problems described above is for performing a predetermined process on text data having a certain unity from the text data. A method of extracting a keyword, the text data having a certain unity, morphological analysis to extract words, the extracted words are evaluated in the text data to evaluate the degree of frequent biased, The gist of the present invention is to extract a word whose evaluation value is equal to or more than a predetermined value as a keyword in the text data.

An abstract sentence generating method according to the present invention, which is performed by using the same technique, is a method for generating an abstract sentence from text data having a fixed unit. The data is subjected to morphological analysis to extract words, the degree to which the extracted words frequently appear in the text data in an uneven manner is evaluated, and words having an evaluation value equal to or greater than a predetermined value are extracted as keywords in the text data. And combining the extracted keywords,
The main point is to generate a summary sentence.

In the keyword extraction technique, words are extracted from text data by using morphological analysis, so that it is not necessary to prepare an extraction thesaurus or the like in advance. In addition, the degree to which the extracted words frequently appear in the text data in a biased manner is evaluated, and the words whose evaluation values are equal to or more than a predetermined value are used as keywords. be able to.
In texts using natural language, it is known that words with high frequency of occurrence are likely to be keywords, but not only high frequency but also the degree to which they appear unevenly is used. Keywords can be extracted except for general-purpose words such as "" and "time". Furthermore, if a summary sentence is generated by combining the keywords thus obtained, a highly accurate summary sentence can be generated very easily. The generation of the summary sentence by combining the keywords is performed, for example, by a morphological analysis from a simple configuration such as generating a fixed form sentence “this text is” + “(extracted keyword group)” + “related” as a summary sentence. From the nouns and verbs obtained in the above, the heights of their associations, and information on the position in the text data (for example, whether or not they existed in the same sentence, etc.), these are appropriately combined to create a summary sentence Various forms such as an output configuration can be considered.

[0012] Here, as text data having a certain unity, data existing in a site connectable via a network, that is, a so-called web page can be assumed. The number of sites connected to a network, for example, the Internet, and the text data having a certain coherence existing there are enormous, and the effect of the present invention on keyword extraction is great.

In extracting words by morphological analysis, extracting words may be limited to a part of words including nouns and sa-variable nouns. In order to reduce the number of words to be examined, preferable. This is because, in the case of Japanese, it is known that nouns and sa-variable nouns play a significant part in meaning. Of course, since the morphological analysis is used, it is easy to extract the verb in the original form.
From among the verbs, basic words called basic words, for example, "run", "drink", "eat" and the like can be further selected and used as keywords.

[0014] The degree to which a word frequently appears in a biased manner can be evaluated by a value obtained by normalizing the number of times the word appears in text data by the amount of the text data. This is known, for example, as TFIDF. T
FIDF is defined by the following equation. In the following equation,
db is a set of symmetrical text data (usually, this corresponds to data to be stored in the database), d is each text constituting the text data, and t is included in the text. Word.

TFIDF = TF (d, t) × Idf (t) (1) where: TF is the number of times the word t appears in the text data d, and Idf is given by the following equation (2). Idf = LOG {DB (db) / f (t, db)} Here, DB (db) is the number of texts in a set of text data, and f (t, db) is a set of text data in the set. Is the number of texts in which the word t appears.

On the other hand, a database can be constructed using the above-described keyword extraction technique. The invention related to the public knowledge of this database classifies a plurality of text data having a certain unit using a keyword,
A method of constructing a database, wherein, for the plurality of text data, a plurality of text data having a predetermined unit are sequentially subjected to morphological analysis to extract words, and the extracted words are included in the text data. Is evaluated, a word having an evaluation value equal to or greater than a predetermined value is extracted as a keyword in the text data, and the plurality of text data is classified by a vector expressed by the extracted keyword. The gist is that a process is performed to construct a database in which the plurality of text data are classified at least by the vector.

According to this database construction method, words are extracted from text data using morphological analysis, so that it is not necessary to prepare an extraction thesaurus or the like in advance. In addition, the degree to which the extracted words frequently appear in the text data in a biased manner is evaluated, and words having an evaluation value equal to or greater than a predetermined value are used as keywords, so that the number of extracted words can be reduced without lowering the accuracy. Can be.
In texts using natural language, it is known that words with high frequency of occurrence are likely to be keywords, but not only high frequency but also the degree to which they appear unevenly is used. Keywords can be extracted except for general-purpose words such as "" and "time".

In addition, a set of target text data is classified by a vector represented by the extracted keyword, and a database classified by at least the vector can be constructed.

[0019] In this database construction method, a category is specified for each of the plurality of text data for each fixed unit, and when constructing the database, the classification using vectors is performed for each category. Is also good. Since similar words may appear in different categories, it is effective to perform classification using a prepared category in order to increase the accuracy of the database. For example, even if the same word "PC" appears frequently, the meaning is completely different for those who search for a site that aims to explain the technical terms of a mail order site. Therefore, it is also effective to classify these by category in advance in terms of subsequent search.

An invention of a search method paired with a database constructed by such a database construction method can be considered. That is, a method of searching for a plurality of text data having a certain unity by using a keyword, wherein, for the plurality of text data,
A plurality of text data having a certain unity,
Morphological analysis to extract words, evaluate the degree to which the extracted words appear frequently and unevenly in the text data, and extract words having an evaluation value equal to or greater than a predetermined value as keywords in the text data; A process of classifying a plurality of text data by a vector represented by the extracted keyword was performed, and a database in which the plurality of text data was classified by at least the vector was constructed, and a keyword to be searched was input. At this point, the gist is that a vector composed of the search keyword is obtained, and matching text data is searched from the database based on the similarity with the vector.

According to such a method, since the comparison is not a comparison between keywords but a vector, it is possible to realize a search in consideration of a region indicated by the entire keyword, that is, a meaningful unity.

Here, as text data having a certain unit, data existing in a site connectable via a network, that is, a so-called web page can be assumed. Since the number of sites connected to a network, for example, the Internet and the text data having a certain unit existing there are enormous, the effect of the present invention on database construction is great.

When extracting words by morphological analysis, the extraction may be performed by limiting the extracted words to some words including nouns and sa-variable nouns. In order to reduce the number of words to be examined, preferable. This is because, in the case of Japanese, it is known that nouns and sa-variable nouns play a significant part in meaning. Of course, since the morphological analysis is used, it is easy to extract the verb in the original form.
From among the verbs, basic words called basic words, for example, "run", "drink", "eat" and the like can be further selected and used as keywords.

The degree to which a word frequently appears in a biased manner can be evaluated by a value obtained by normalizing the number of times the word appears in text data by the amount of the text data. This is known, for example, as TFIDF (described above).

As an invention corresponding to such a keyword extraction method, a summary generation method, a database construction method, and a search method, there are an apparatus and a program for implementing these methods, and a recording medium on which the program is recorded. Of course you get.

[0026]

Another embodiment of the present invention The technique relating to keyword extraction of the present invention can be used for, for example, translation. For translation, it is effective to create a database of translation examples.
This is because it can be applied to such a database search.

[0027]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below based on examples. (1) Configuration of Embodiment: First, the configuration of the embodiment will be described with reference to FIG. FIG. 1 is a schematic configuration diagram showing a system for constructing a database according to the present embodiment. This system is realized as a database server 200 connected to a large-scale network 100 such as the Internet. An enormous number of servers 300, 310, 320,... Are connected to the network 100.
A large number of web pages WP are stored in storage devices in these servers 300. Different addresses (UR
The web page given L) is herein referred to as a group of text data. These text data may include meta tags and the like. Also, these web pages are usually the top page (hereinafter referred to as “cover”) FP directly pointed to by the address.
And a lower page that can be called from this display FP (hereinafter referred to as
BD (referred to as “body” for convenience) (see FIG. 5 and described later in detail). Of course, there may be a web page composed of a single page or a page constructed with complicated links, but for convenience of explanation, the front page FP
With the configuration of the web page consisting of
The following is described.

The database server 200 includes a network interface (NT-I / F) 210 for controlling data exchange with the network 100, a CPU 220 for processing, a ROM 230 for storing processing programs and fixed data, and a work area. RAM 24
0, a timer 250 for managing time, a database (DB) 260 for storing various data described later, a hard disk 270 for storing a Japanese dictionary, and the like.
Although the database 260 is actually stored in a storage device such as a hard disk, it is assumed here that the database 260 is handled as an independent device for convenience of explanation.

In this system, text data in a site provided in a large number of servers 300,... Made public via the network 100 are classified and made public in a searchable manner. For this purpose, some processes are performed according to the procedure shown in FIG. This procedure is configured as follows.
First, data is collected (step S10). Thereafter, a process of extracting and classifying the collected text data by keywords is performed (step S20), and a process of constructing a database using the obtained keywords is performed (step S30). It is made public and the user can freely access it (step S40). Thus, this database 260 can be used by anyone or only registered members.

(2) Beta Base Construction Process: The database construction process described as steps S10 to S30 will be described in detail below. The database construction processing of the present embodiment includes keyword extraction processing and processing of classifying text data using the extracted keywords and thereby constructing a database. As shown in FIG. 4, when the database construction process is started, first, a process of collecting data by the traveling robot is performed (step S11). This process is a process of outputting addresses specifying sites in the servers 300,... Via the network 100, and collecting text data from those sites. When the network 100 is the Internet, the circulating sites are sequentially specified by an address called an IP address. In the case of an IP address, the assignment of the global address is
Since it is determined locally to some extent, it is possible to limit the target of the patrol engine to, for example, Japan or to Japan and the United States. When acquiring address information called a URL from a site designated by an IP address, an address whose address starts with "http: //" is a hypertext, and constitutes a so-called web page. It is also possible to acquire text data only for pages having.

The URL of the hypertext obtained first by specifying a certain IP address can be handled as the front page FP of the text data. For example, as shown in FIG. 5, the URL obtained by specifying a certain IP address is “http://www.AAA.xx.jp/INDEX.H
In the case of “TML”, this URL is used as the front page FP. When data is read from this front page FP, the page contains the address of another page linked from this page FP. In the example shown in FIG. 5, "http://www.AAA.xx.Jp/BBB/INDE
X.HTML "or" http://www.AAA.xx.jp/CCC/INDEX.HTML "
Etc. correspond to this. The patrol engine also obtains all the text data of such links. However, it does not follow links to other web pages. That is, text data in the same IP address is collected together with the URL.

Next, the text data thus obtained is subjected to morphological analysis to extract words (step S21). Since the morphological analysis is well known as a technique of Japanese analysis, detailed description is omitted, but as shown in FIG. 6, a Japanese dictionary JD prepared in advance in a storage device such as a hard disk 270, particularly a so-called reverse lookup dictionary Using Ijd, the obtained text data is analyzed, and words constituting each document are determined by morphological analysis. For example, in the example shown in FIG. 5, for a sentence "A car named DD is a sedan with an emphasis on quality."
D, “to,” “to,” “to,” “to,” “to,” “car.
“Ha” “quality” “weight” “weight” “do” “do” “ta”
Words such as "sedan", "de", "a", "is", and "a" can be extracted. Here, "I" and "U"
The kana single sounds such as "a", "shi" and "ta" are also extracted as words, such as the stem "i" of "say" or "u" of "ru" (sell). This is because they may appear in the sentence.

The dictionary Ijd stores these words together with their grammatical information. Then, the extracted words are arranged next according to the grammatical information, and a process for finding an array that does not break down is performed. For such an analysis, for example, a method such as a multiple phrase longest matching method or a minimum cost method is known, and a test is performed to determine which combination of predetermined words is most likely to be Japanese. For example, taking "quality" as an example, under the rule that the association between an independent word and an adjunct (particle) is more preferable than an independent word + an independent word + adjunct (particle), the "product" (independent "Quality" (independent word / noun) + "wo" (adjunct / particle) is better than "quality" (independent word / noun) + "wo" (adjunct / particle) They judge that it is likely to be Japanese.

After performing the morphological analysis in this way, based on the part of speech information of the obtained words, only words corresponding to the noun and the noun are extracted. Of course, the original verb, adverb, adjective, etc. may be extracted. How to extract words depends on the type of text data to be classified.For example, in ordinary text data, extraction is performed mainly on nouns, and text data on appreciating literary works and artworks is not It is also preferable to extract mainly adjectives and the like. For text data on sports, it is also possible to extract verbs.

In this example, morphological analysis is performed using information on grammatical terms and their associations.
If the words to be extracted are limited to kanji idioms and katakana, simple kanji character strings and katakana character strings are extracted from the text data as words, and a simple judgment is made as to whether these words are listed in the noun dictionary , It is also possible to extract words.

The words extracted by the morphological analysis are registered in the temporary registration database. Therefore, the following describes the configuration of the temporary registration database. This temporary registration database is different from the database obtained finally, in which the addresses of text data collected by the traveling engine and the words extracted from the text data are temporarily registered for keyword extraction processing. It is a database to keep. The temporary registration database includes various tables TB described below.

The temporary registration database has the structure shown in FIG. In other words, this database is "Host"
The table is composed of four tables “Page”, “keyword”, and “word”. The tables “Host” and “Page” are based on ID numbers, and the tables “Page” and “keyword” are based on PageID, and the tables “Keyword”
And “word” are related by WordID.

FIGS. 8 to 11 show the details of these various tables TB. The “Host” table HTB is
This is a table including IDs and “HostName”, and different IDs are associated with different web pages. Therefore, text data is classified in units of sites having IDs in the table HTB (usually sites having a domain name corresponding to one IP address).

As shown in FIG. 9, the “Page” table PTB is a table in which “Host ID”, “Page ID” and “address” are associated with each other. Of these, “K HostID” is the same as the ID in the “Host” table HTB. Addresses included in the same site are given the same “HostID”, and “PageID” is given to pages below them. “PageID” does not allow duplication, and differs for each address. Therefore, in the example shown in FIG. 9, the web page (“HostID” = 22) represented by “www.AAA.xx.jp” includes “www.AAA.xx.j”.
p / CCC / INDEX.HTML '' or `` www.AAA.xx.jp/DDD/power.HT
ML "and" www.AAA.xx.jp/EEE/Keep.HTML ". "Page
The “ID” also does not allow duplication, and a unique number is assigned to all pages. It should be noted that “page” in these descriptions does not mean a page as a print unit but means a group of text data to which one URL is assigned. Thus, given a single URL,
This is one page, whether it is a page made up of very few text data or a page made up of tens of pages of text data if printed.

The "Page" table PTB
In the present embodiment, is configured to include all of a plurality of “HostIDs”, but may be provided for each “HostID”. Similarly, "keyword" described below
The table KTB and the “word” table WTB are also
"ge" or "keyword".

The "keyword" table KTB is shown in FIG.
Is a table in which “WordID”, “PageID”, and “Cost” are associated with each other. Among them, “WordID” is an ID assigned to a word previously extracted by morphological analysis, and a unique value is assigned to each word. Then, how many times each word appears in a page having one “PageID” is counted, and this is stored in “Cost”.

The “word” table WTB stores “WordI”
It is a table that stores "D", "word", and "F value" in association with each other. That is, the “keyword” table KTB
Gives the number of appearances "Cost" in the page for each word
Is required, so this is a non-zero page, ie,
The number of pages on which the word appears is determined for each word ("WordI
D ”). Then, the word t and its cumulative value F
(T, db) is obtained for each site and is stored. In the example shown in FIG. 11, the number of pages 306 where the word "car" appears in this site, and the number of pages where general terms such as "features" appear are as large as 124
0, and so on.

Thus, the processing up to step S22 shown in FIG. 4 is completed, and then the frequency evaluation processing is performed (step S23). This processing is, specifically, the TF described above.
A process for determining the value of IDF is performed. The value of TFIDF is
Equations (1) and (2) are obtained from equations (1) and (2).
When applied to the form of a web page possessed by each site, TFIDF = TF (d, t) × Idf (t) (1) where: TF is a page specified by one URL (text BD) d , The number of times the word t appears. Therefore, in the cases shown in FIGS. 6 to 11,
Taking "car" as an example, "WordID" = 3,
An address with “PageID” = 4 “www.AAA.xx.jp/
In the "CCC / INDEX.HTML", the word "car" appears twice, and the appearance frequency TF (d, "car") normalized by the total number of words 807001 is Becomes

On the other hand, the Idf is calculated according to FIG. 11 and the following equation (2). Idf = LOG {DB (db) / F (t, db)} (2) where DB (db) is the number of all pages existing in a specific site, and therefore, in the example shown in FIG. , The same "H
ostID ", in this example,
36456. On the other hand, F (t, db) corresponds to FIG.
Is the F value shown in FIG. When the formula (2) is calculated for “car”, Idf = LOG (36456/306) = LOG (1257) = 4.780760 Therefore, TFIDF = TF × Idf = 0.0000024 × 4.77802760 = 0.00000115 Become.

FIG. 12 shows the result of performing the above calculation for the words “car”, “feature”, “next generation”, and “energy” with “PageID” = 4. As a result, if only the number of appearances (value of TF) is satisfied, “feature”> “vehicle” = “next generation”> “energy”, but in the TFIDF value, “next generation”> “energy” It can be seen that the order is "feature">"car">"energy".

Next, an average vector is calculated (FIG. 4, step S24). The above-mentioned calculation of TFIDF is
This is performed for one “PageID”. That is,
Since one site is usually composed of a plurality of pages, if the above operation is performed for each page (text data having a unique URL under one IP address), a TFIDF value is obtained for each page be able to. Therefore, an average vector is obtained by averaging these TFIDF values. That is, assuming that the number of pages existing in one site is N and the TFIDF value of each page is TFIDFi (i = 1, 2,... N), the average vector TFIDFav is: TFIDFav = (TFIDF1 + TFIDF2 +.
TFIDFN) / N. Thus, the TFIDF value at that site for one word was determined.

Thus, for each keyword, the average TF
At the stage when the IDF value is obtained, only words having a TFIDF value of a predetermined value, for example, a value of 0.00001 or more, are extracted as keywords. Next, a process is performed to calculate a vector Ba for this site (www.AAA.xx.jp) and to register this in the database as a keyword for this site (step S31). Ba = (b1, b2... Bm) b1, b2.
001 or more words and their average TFIDF values. After obtaining the vector Ba for one word in this way, the above processing is repeated for all words appearing on all pages of all sites. As a result, information on a huge number of sites collected by the traveling engine is accumulated as a set of vectors Ba. This is Database 2
Equivalent to 60.

In the above-described vector calculation and registration processing, the vector Ba may be composed of only words whose average TFIDF is equal to or greater than a predetermined value, or composed of all words prepared in the dictionary as elements. Is also good. In this case, TFI
The TFIDF value for a word whose DF value is equal to or less than a predetermined value is:
Approximate value 0. In any case, the number of elements in the vector decreases or the number of elements having the value 0 increases, so that the calculation can be easily performed.

When the database 260 is completed by the above processing, the database is released to the outside and used for free use or for registered members. At this time, a configuration in which the database is directly accessed is also possible. However, in order to enable access from an unspecified number of clients via the network 100, for example, a site having a CGI for accessing the database 260 may be configured as follows: Prepared in the server 200, the client:
Usually, the database 260 can be accessed from a so-called browser via the network 100. Therefore, a method of searching for a web page using a database will be described next. FIG. 13 is a flowchart showing a process at the time of search. First, the client that starts the search accesses a site prepared for the search (step S40).
0). As a result, a search screen as shown in FIG. 14 is displayed.

Therefore, the client inputs the search content as a sentence in Japanese into the keyword entry box KB prepared on this screen (step S410).
For example, as shown in FIG. 14A, a character string such as "next generation" is input into a box TB for inputting a character string. At this time, as shown in the figure, a search field or the like may be specified together. At this time, if it is necessary to perform a refined search, the interface shown in FIG. 14A may be displayed again to narrow down sequentially, or a comma (,) such as “next generation, car” may be used.
May be used to input a plurality of words. Or,
As illustrated in FIG. 14 (B), a natural sentence such as “about the next generation car” may be input. At this time, in parallel with the input of the search sentence, it is monitored whether the "search" button BB is pressed (step S420). When the search button is pressed, the input word or sentence is read, and FIG. 14), the word and the field are extracted, and in the case of the input shown in FIG. 14 (B), the sentence is morphologically analyzed, and the word is extracted anyway (step). S430). When a word is extracted by morphological analysis, the word may be extracted by limiting it to nouns or sa-variant nouns, or may be extracted including other parts of speech. FIG. 14B also schematically shows how words are extracted from a search sentence.

After extracting words, or words and fields,
With respect to the obtained s words D1, D2,..., Ds, processing for obtaining the vector Bs is performed (step S44).
0), a process of searching the database 260 for a site having a vector closest to the vector Bs (step S450). That is, as schematically shown in FIG. 15, each site has a number of words as elements,
Since the vector 260 is stored in the database 260 as a vector composed of a set of words weighted by the IDF value, the vector composed of the search keyword obtained from the given sentence and the vector registered in the database 260 The similarity is determined, and the search results are output in order from the site having the most similar vector (step S470). The output search results are sent to the client via the network 100 and displayed on the screen of the client machine.

According to such a method, sites are classified using information (TFFID value) indicating how skewed a word appears in a page constituting the site, and this is stored in the database 260. , Registered as a set of words having a TFIDF value equal to or greater than a predetermined value, and looking at the similarity between this data and the vector of words given as a keyword for search, it is not necessary to simply look for a match between keywords. Search that captures the unique features of the system.

Next, a second embodiment of the present invention will be described. In the second embodiment, substantially the same processing as that of the first embodiment is performed, but when constructing a database, first, as a preliminary processing, a processing of assigning a classification by manual processing to some representative sites is performed. Perform it together. That is, the patrol engine collects information on, for example, thousands of sites, extracts words from text data present on the sites, calculates TFIDF values, and obtains vectors. A registrant refers to the front page and performs a process of assigning a classification appropriate to the front page. That is, in step S31 shown in FIG. 4, when registering a vector composed of words having a TFIDF value of a predetermined value or more, a classification item is added. Classification items are "mail order", "hobby", "politics"
Various classifications such as "economy" can be applied. Of course, an industrial classification or the like may be used.

In this case, as illustrated in FIG. 16, the vectors of many sites included in the classification given manually exist with a certain extent. Therefore,
The center of this spread is defined as a vector B representing such a classification.
Are defined as C1, BC2,. Also, the degree of variation (dispersion) with respect to the center can be determined from the spread of the vector of the processed site. By performing such processing in advance, when text data of all sites on the Internet is next collected by the traveling engine, the classification of the sites can be easily determined from the obtained vectors. The database 260 may register information of each site without having a specific classification as in the first embodiment, but if a classification is given, the information may be presented in a form such as a table of contents. Also becomes possible.

According to this embodiment, since the center and the spread of the classification can be defined in a vector manner, the text data of the new site is analyzed, so that the classification of the site can be easily determined. Can be. If there is a site that cannot be classified into any of them, a warning may be given to the operator of the server 200 to that effect, and a process of adding a new classification may be performed.

When the database 260 with such classification is prepared, when the client performs a search, it is possible to narrow down the search range by first specifying this classification. Since there are many sites and the like on the Internet, it is effective to perform a search by giving a classification to improve the efficiency of the search.

Next, a third embodiment of the present invention will be described. The third embodiment is a summary sentence generation device that generates a summary sentence from given text data. This summary sentence generation device is provided in the server 200 of the first embodiment, and generates a summary sentence using the database generation processing described in the first embodiment. That is, as shown in FIG. 17, after the registration in the database 260 is completed (FIG. 4, step S
31) Then, a keyword registered for one site is read (step S500), and a process of extracting five words L having the highest TFIDF values from the keyword is performed (step S510). Then, arranging these words L1, L2, ... L5, "This site is L
1, L2, L3, L4 and L5. Is performed (step S520). Since this sentence can be regarded as the shortest sentence of the contents of this site, it is registered in the database 260 (step S530). Thereafter, when a search is performed from the client and a site similar to the content specified from the search keyword group is output, this sentence is output as a summary sentence together with the URL.

According to this embodiment, it is possible to easily generate a summary that expresses the contents of the site most simply,
It can be used as extremely effective information to know the contents of the searched site. In this example, it is assumed that only nouns and sa-variant nouns are registered as keywords, but verbs and adjectives are registered as keywords, and relationships between the words, for example, appear on the same page. If the information such as whether or not is stored, a certain sentence may be generated using morphological analysis. In this case, for example, if the adjective a1 and the verb V1 appear on one page centering on the noun L1, a sentence such as "This site relates to the fact that a1L1 is V1" is generated. Can be. Of course, between the noun L1 and the verb V1, there are those that can be “subject + predicate” and those that can be “object + predicate”, and such information is prepared in a dictionary or the like in advance. Therefore, it can be said that the noun L1 and the verb V1 are tested to generate a sentence "This site relates to a1L1 about V1."
At the end of the sentence, if V1 is a sa noun, it may be generated as a natural Japanese language such as "to do V1".

Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments at all, and may be implemented in various other forms without departing from the gist of the present invention. Obviously you can get it.
For example, the present invention may be realized as an apparatus or a method that only constructs a database, or may be realized as an apparatus or a method that only extracts keywords. Further, the present invention can be applied to a translation device. For translation, simply trying to convert between languages using grammar information does not work (the necessary rules become infinitely large), but rather prepares abundant examples, finds examples that match the translation, It is known that translating in such a way as to apply this can result in a semantically accurate translation. Therefore, it is possible to use the present invention by applying the present invention to given text data, extracting a keyword, and using this to specify an example.

[Brief description of the drawings]

FIG. 1 is a schematic configuration diagram showing an overall configuration in each embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a database server 200.

FIG. 3 is an explanatory view schematically showing a process in an example.

FIG. 4 is a flowchart showing a database construction process performed by a database server 200;

FIG. 5 is an explanatory diagram illustrating a state of a data link on a web page.

FIG. 6 is an explanatory diagram illustrating morphological analysis on text data.

FIG. 7 is an explanatory diagram showing a configuration of a temporary registration database.

FIG. 8 is an explanatory diagram illustrating an example of a “Host” table HTB.

FIG. 9 is an explanatory diagram showing an example of a “Page” table PTB.

FIG. 10 is an explanatory diagram illustrating an example of a “keyword” table KTB.

FIG. 11 is an explanatory diagram illustrating an example of a “word” table WTB.

FIG. 12 is an explanatory diagram showing a calculation example of a TFIDF value.

FIG. 13 is a flowchart illustrating a search process according to the embodiment.

FIG. 14 is an explanatory diagram illustrating an example of a search screen.

FIG. 15 is an explanatory diagram schematically showing a state of similarity determination in a search.

FIG. 16 is an explanatory diagram schematically showing a relationship between a classification and a vector.

FIG. 17 is a flowchart illustrating a summary sentence generation process.

FIG. 18 is an explanatory diagram showing a conventional method for determining a similarity of data by obtaining a vector from a keyword.

[Explanation of symbols]

 100 network 200 database server 220 CPU 230 ROM 240 RAM 250 timer 260 database 270 hard disk 300, 310, 320 server

Claims (18)

[Claims] 1. A method for extracting a keyword for performing a predetermined process on text data having a certain unit from text data having a certain unit, wherein the text data having the certain unit is subjected to morphological analysis to determine a word. A keyword extracting method for extracting the extracted words, evaluating a degree of the occurrence of the extracted words in the text data in a biased manner, and extracting words having the evaluation value equal to or more than a predetermined value as keywords in the text data. 2. The extraction method according to claim 1, wherein the text data having the predetermined unit is data existing in a site connectable via a network. 3. The extraction method according to claim 1, wherein in extracting the words by the morphological analysis, the extracted words are extracted by limiting the extracted words to a part of words including nouns and sa-variable nouns. 4. The method according to claim 1, wherein the degree to which the word frequently appears in a biased manner is evaluated by a value obtained by normalizing the number of times the word appears in the text data by the amount of the text data. The extraction method according to any of the above. 5. A method for classifying a plurality of text data having a certain unit using a keyword and constructing a database, wherein the plurality of text data having the certain unit are sequentially arranged for the plurality of text data. Text data of
Morphological analysis to extract words, evaluate the degree to which the extracted words appear frequently and unevenly in the text data, and extract words having an evaluation value equal to or greater than a predetermined value as keywords in the text data; A database construction method for performing a process of classifying a plurality of text data by a vector represented by the extracted keyword, and constructing a database in which the plurality of text data is classified by at least the vector. 6. The database construction method according to claim 5, wherein the text data having a predetermined unit is data existing in a site connectable via a network. 7. The database construction method according to claim 5, wherein in extracting the words by the morphological analysis, the extracted words are extracted by limiting the extracted words to a part of words including nouns and sa-variable nouns. 8. The method according to claim 5, wherein the degree to which the word appears in a biased manner is evaluated by a value obtained by normalizing the number of times the word appears in the text data by the amount of the text data. The database construction method according to any of the above. 9. The method according to claim 5, wherein a category is specified for each of the plurality of pieces of text data in a certain unit, and the plurality of text data is represented by a vector when the database is constructed. A method for constructing a database for performing the classification by the category. 10. A method for generating a summary sentence from text data having a certain unity, wherein the text data having the certain unity is morphologically analyzed to extract words, and the extracted words are Evaluating the degree of frequent occurrence in the text data, extracting words whose evaluation value is equal to or greater than a predetermined value as keywords in the text data, and combining the extracted keywords to generate a summary sentence Method. 11. A method for searching for a plurality of text data having a certain unit using a keyword, wherein the plurality of text data having a certain unit are sequentially searched for the plurality of text data.
Morphological analysis to extract words, evaluate the degree to which the extracted words appear frequently and unevenly in the text data, and extract words having an evaluation value equal to or greater than a predetermined value as keywords in the text data; A process of classifying a plurality of text data by a vector represented by the extracted keyword was performed, and a database in which the plurality of text data was classified by at least the vector was constructed, and a keyword to be searched was input. A search method for obtaining a vector including the search keyword, and searching the database for matching text data based on the degree of similarity with the vector. 12. An apparatus for extracting a keyword for performing a predetermined process on text data having a certain unit from text data having a certain unit, wherein the text data having the certain unit is subjected to morphological analysis to convert a word. A morphological analysis unit to be extracted; a frequency evaluation unit that evaluates the degree to which the extracted words appear in the text data in a biased manner; and a word whose evaluation value is equal to or greater than a predetermined value is extracted as a keyword in the text data. A keyword extraction device comprising keyword extraction means. 13. An apparatus for classifying a plurality of text data having a certain unit using a keyword and constructing a database, wherein the plurality of text data having the certain unit is
Morphological analysis means for extracting words by morphological analysis; frequency evaluation means for evaluating the degree to which the extracted words appear in the text data in a biased manner; A keyword extracting unit that extracts a plurality of pieces of text data as a keyword, and a classifying unit that classifies the plurality of pieces of text data by a vector represented by the extracted keyword. By performing the processing according to, the plurality of text data,
A database construction device for constructing a database classified by at least the vector. 14. An apparatus for generating a summary sentence from text data having a certain unity, comprising: a morphological analysis unit for morphologically analyzing the text data having a certain unity to extract words; Frequency evaluation means for evaluating the degree to which words frequently appear unbalanced in the text data; keyword extraction means for extracting words whose evaluation value is equal to or greater than a predetermined value as keywords in the text data; A summary sentence generation apparatus comprising: a sentence generation unit that generates a summary sentence by combining them. 15. An apparatus for searching a plurality of text data having a certain unit using a keyword, wherein the plurality of text data having the certain unit are sequentially searched for the plurality of text data.
Morphological analysis to extract words, evaluate the degree to which the extracted words appear frequently and unevenly in the text data, and extract words having an evaluation value equal to or greater than a predetermined value as keywords in the text data; A process for classifying a plurality of text data by a vector represented by the extracted keyword; a database storage unit for storing a database obtained by classifying the plurality of text data by at least the vector; and a keyword to be searched. A search device comprising: vector calculation means for obtaining a vector consisting of the search keyword when input; and search means for searching for matching text data from the database based on the similarity with the vector. 16. A program for causing a computer to perform a process of extracting a keyword for performing a predetermined process on text data having a certain unit from text data having a certain unit, comprising: A function of extracting words by performing morphological analysis; a function of evaluating the degree to which the extracted words appear frequently in the text data in a biased manner; A program to realize the function of extracting as. 17. A program for causing a computer to perform a process of classifying a plurality of text data having a certain unit using a keyword and constructing a database, wherein the plurality of text data are sequentially processed with respect to the plurality of text data. Multiple text data with a certain unity
Morphological analysis to extract words, evaluate the degree to which the extracted words appear frequently and unevenly in the text data, and extract words having an evaluation value equal to or greater than a predetermined value as keywords in the text data; A program for realizing a function of performing a process of classifying a plurality of text data by a vector represented by the extracted keyword, and a function of constructing a database in which the plurality of text data is classified at least by the vector. 18. A program for causing a computer to perform a process of generating a summary sentence from text data having a certain unity, a function of morphologically analyzing the text data having a certain unity and extracting words. A function of evaluating the degree to which the extracted words appear in the text data in a biased manner, a function of extracting words having an evaluation value equal to or greater than a predetermined value as keywords in the text data, A program for realizing the function of combining and generating a summary sentence.