JP5108660B2 - Information collection method, apparatus, and program - Google Patents

Description

  The present invention relates to an information collection method, apparatus, and program for collecting information described in a Web document.

  Conventionally, when it is desired to obtain information from a Web document regarding a specific item, the Web document has to be browsed and collected.

Therefore, when it is desired to examine information related to a personal computer, according to the technique described in Non-Patent Document 1, the user can obtain information on personal computers for each of various manufacturers and sales companies based on information displayed in a list or the like. Can be explored.
[Online], [Search June 18, 2008], Internet <URL: http: // kakaku. com / pc / desktop-pc />

  However, even with the technique described in Non-Patent Document 1, since information relating to a personal computer is manually collected, accumulated, and re-edited, the time and labor required for the work become enormous. Further, the technique described in Non-Patent Document 1 is provided for specific products, and there is a limit in coverage of stored information when trying to examine products that are not handled or products other than products.

Therefore, the present invention is proposed in consideration of such circumstances, and automatically displays information on a common item distributed on a Web page and information related to its attribute and attribute value. An object of the present invention is to provide an information collection method, apparatus, and program to be collected.

  In order to achieve the above object, the present inventor has found a mechanism for automatically collecting information on items from a plurality of Web documents, and has come up with the present invention.

  The information collection method according to the present invention automatically collects information on items from a Web document by extracting information having a relationship between items, attributes, and attribute values based on tags included in the Web document. is there.

(1) an information collecting apparatus extracting information in a table format or a database format from a Web document accessible via a communication network based on a tag included in the Web document;
From the extracted information in the table format or database format, based on the dependency relationship between the information indicated by the tag, the relationship between the attribute dependent on the predetermined item and the attribute value indicating the content of the attribute Extracting information having,
And a step of associating and storing in the storage means information having a relationship between the extracted item, attribute and attribute value.

  According to the configuration of the invention described in (1), information in a table format or a database format is extracted based on a tag included in a Web document, and each piece of information indicated by the tag is extracted from the extracted table format or database format information. Based on the subordinate relationship, information having a relationship between an attribute subordinate to a predetermined item and an attribute value indicating the content of the attribute is extracted.

  This makes it possible to automatically collect information having a relationship between items, attributes, and attribute values from a plurality of Web documents.

  Here, the information in the table format includes information indicated in plain text expressed in the same way as the table by comma separation, space separation, or the like.

  In addition, information that is commonly subordinated in a larger number of Web documents with respect to a predetermined item may be extracted as an attribute and an attribute value with a greater weight. In addition, information that does not commonly depend on a predetermined item in a number of Web documents exceeding a predetermined threshold may not be extracted as the attribute and the attribute value.

  Here, the attribute and the attribute value extracted and stored can be used for suggesting a search query when searching for a Web document related to an item.

  (2) The method according to (1), wherein the extracting step extracts, as the item, information located immediately above, immediately below, or immediately to the left of the tabular information.

  According to the configuration of the invention described in (2), information located immediately above, directly below, or immediately to the left of the tabular information is extracted as the item.

  This makes it possible to extract information located at a position where a title of tabular information is often displayed in a Web document as an item for the tabular information.

  Here, as described above, the accuracy may be increased by multiplying the weight according to the frequency having the relationship or not performing the extraction until the frequency reaches a predetermined threshold.

(3) In the extracting step, the information located in the uppermost row or the leftmost column is extracted as the attribute in the tabular information, and the information located below or on the right is extracted as the attribute value (1) or ( The method according to 2).

  According to the configuration of the invention described in (3), in the tabular information, the information located in the upper row or the left column is extracted as the attribute, and the information located below or to the right is extracted as the attribute value.

  As a result, information located at positions where attributes of tabular information are often displayed in Web documents are extracted as attributes, and information located at positions where the contents of the attributes are often displayed are extracted as attribute values. can do.

  Here, as described above, the accuracy may be increased by multiplying the weight according to the frequency having the relationship or not performing the extraction until the frequency reaches a predetermined threshold.

  (4) In the extracting step, information located immediately to the left of the database format information is used as an attribute, and the database format information is extracted as an attribute value thereof. the method of.

  According to the configuration of the invention described in (4), information located immediately to the left of the database format information is used as an attribute, and the database format information is extracted as its attribute value.

  As a result, information located in a position where the attribute of the information in the database format is often displayed in the Web document is used as an attribute, and information in the database format in which the content of the attribute is often displayed is extracted as an attribute value. be able to.

  Here, as described above, the accuracy may be increased by multiplying the weight according to the frequency having the relationship or not performing the extraction until the frequency reaches a predetermined threshold.

  (5) The tag based on extracting information in the table format or the database format is an HTML (HyperText Markup Language) tag that forms a pull-down list, a tag included in an XML (Extensible Markup Language) document, or an HTML table tag The method according to any one of (1) to (4).

  According to the configuration of the invention described in (5), since it is determined as information in a table format or a database format based on an HTML tag forming a pull-down list, a tag included in an XML document, or an HTML table tag, By determining the matching of the included tags, information in a table format or a database format can be automatically extracted.

  (6) A program that causes a computer to execute the method according to any one of (1) to (5).

(7) Information group extracting means for extracting information in a table format or a database format from a Web document accessible via a communication network based on a tag included in the Web document;
From the extracted information in the table format or database format, based on the dependency relationship between the information indicated by the tag, the relationship between the attribute dependent on the predetermined item and the attribute value indicating the content of the attribute An information collection apparatus comprising: an attribute relationship extraction unit that extracts information stored therein and associates information having a relationship between the extracted item, attribute, and attribute value and stores the information in a storage unit.

According to the present invention, by extracting information having a relationship between items, attributes, and attribute values based on tags included in the Web document, common items existing on the Web page and their attributes are distributed. And information related to the attribute value can be automatically collected.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings. This is merely an example, and the technical scope of the present invention is not limited to this .

[Overall configuration of information collection device and related elements]
In FIG. 1, an information collection device 1 is connected to a plurality of Web server devices 2 through the Internet N as a communication network. It does not matter whether the connection with the Internet N is wired or wireless.

The information collection device 1 acquires Web documents from a plurality of Web server devices 2. Each Web server device 2 provides various Web documents in response to requests from the information collection device 1 .

[Functional configuration of information collection device]
FIG. 2 is a diagram illustrating an outline of a functional configuration of the information collecting apparatus 1 according to the present embodiment. The information collection device 1 includes a Web document storage unit 11, an information group extraction unit 12, and an attribute relationship extraction unit 13. In addition, it has a Web document DB 15, an information group storage unit 16, and an attribute relationship DB 17 (DB is an abbreviation for database).

  The Web document storage unit 11 acquires a Web document from the Web server device 2 and stores it in the Web document DB 15. The information group extraction unit 12 reads the Web document stored in the Web document DB 15, extracts information in a table format or a database format based on the tag included in the read Web document, and stores the information in the information group storage unit 16. The attribute relationship extraction unit 13 reads information in a table format or a database format stored in the information group storage unit 16, and based on the tag included in the read information, the relationship between items, attributes, and attribute values included in the information Is extracted, and the extracted information is registered in the attribute relation DB 17. The Web document storage unit 11, the information group extraction unit 12, and the attribute relationship extraction unit 13 are realized by a computer executing a program.

The Web document DB 15, the information group storage unit 16, and the attribute relationship DB 17 are provided in one area of the storage device 410 included in hardware described later .

[Configuration of various databases and related elements]
FIG. 3 is a diagram showing an outline of the configuration of the Web document DB 15, the information group storage unit 16, the attribute relationship DB 17, and related elements.

As shown in FIG. 3 (a), Web documents DB15 includes a document ID, a source code is a description of the communication address and the Web documents on the Internet N such as URL of the Web document that is delivered over the Internet N Are stored in association with each other.

  As shown in FIGS. 3B and 3C, the information group storage unit 16 stores information in the table format or database format extracted from the source code of the Web document together with tags constituting the table format or database format. Stored.

As shown in FIG. 3E, the attribute relationship DB 17 stores information having a relationship between items, attributes, and attribute values extracted from information in a table format or a database format.

  As shown in FIG. 3D, the present embodiment includes an attribute dictionary 14 that is used when extracting information having a relationship between items, attributes, and attribute values from information in a table format or a database format.

  FIGS. 3B and 3C are examples of information in a table format or database format extracted from a Web document. For example, the Web page (Web document) of the PC sales company A includes information in the table format or database format shown in FIG. 3B, and the Web page (Web document) of the PC sales company B is shown in FIG. It is assumed that the information in the table format or database format shown in c) is included.

  The information may be variously described as a group of HTML forming a pull-down list, described as an XML document, or described as a group of HTML forming a table with a table tag.

When it is described as a group of HTML that forms a pull-down list, for example, it is possible to determine the start tag and end tag of the <Select> tag and extract the start tag and end tag and the contents of the elements in between Conceivable.

  If the XML instance is described as an XML document, the XML instance has a hierarchical structure of information. For example, it is considered to extract the contents of an element between the top start tag and end tag of the XML instance. It is done.

If the table tag is described as a group of HTML forming a table, for example, the start tag and end tag of the <Table> tag are determined, and the start tag and end tag and the contents of the elements between them are determined. Can be considered.

  In the present embodiment, in the information of FIG. 3B, the table tag describes the item “note PC” in the first column, and “CPU” and “clock” as the attributes of the notebook PC in the first row. , “CPU xxx” and “CPU yyy” are described as attribute values of the attribute “CPU”, and “1.5 GHz” and “2.0 GHz” are described as attribute values of the attribute “clock”.

  When the relationship between items, attributes, and attribute values is accumulated in a database server or the like that stores the data that is the basis of these Web documents and is acquired using a program such as CGI, Information having these relationships is collected by executing a program such as CGI.

  In the present embodiment, the information in FIG. 3C is assumed to be formed by a table tag, and an item is included in a part of the <Caption> tag element content described as the <Table> tag element content. “Note PC” is described, and “CPU” and “Clock” are described as attributes of the item “Note PC” in the first row of the table immediately after that, and thereafter, “CPU” is set as an attribute value of the attribute “CPU”. “zzzz” and “CPU ppp” are described, and “800 MHz” and “3.2 GHz” are described as attribute values of the attribute “clock”.

FIG. 3D is an example of the attribute dictionary 14 used by the attribute relationship extraction unit. The attribute dictionary 14 is stored in a storage device 410 (see FIG. 4) included in hardware described later. In the present embodiment, the attribute dictionary 14 associates items with attributes. For example, in addition to “notebook personal computer”, an item such as “notebook PC” is registered in the item. On the other hand, “CPU”, “HDD”, “battery”, “price”, and the like are registered as attributes related to the items. Furthermore, a lower attribute (attribute 2) related to the attribute (attribute 1) is registered. For example, for attribute 1 “CPU”, attribute 2 “clock”, “cache”, and the like are registered.

FIG. 3E shows an example of information having a relationship between items, attributes, and attribute values stored in the attribute relationship DB 17 . For each attribute relationship ID that is a unique number, an item, an attribute, and an attribute value are stored in association with each other. For example, an item “notebook personal computer” is stored assuming that the attribute value of attribute 1 “CPU” is “CPU xxx” and the attribute value of attribute 2 “clock” is “1.5 GHz” .

[Hardware configuration diagram of information collection device]
FIG. 4 is a diagram illustrating a hardware configuration of the information collecting apparatus 1 according to the present embodiment.
The information collection device 1 includes a CPU (Central Processing Unit) 310 (a plurality of CPUs such as a CPU 320 may be added in a multiprocessor configuration), a bus line 200, a communication I / F (I / F: An interface) 330, a main memory 340, a BIOS (Basic Input Output System) 350, an I / O controller 360, a hard disk 370, an optical disk drive 380, and a semiconductor memory 390. The hard disk 370, the optical disk drive 380, and the semiconductor memory 390 are collectively referred to as a storage device 410.

  The control unit 300 is a part that controls the information collecting apparatus 1 in an integrated manner, and appropriately reads and executes various programs stored in the hard disk 370 (described later), thereby cooperating with the above-described hardware. Various functions related to are realized.

  The communication I / F 330 is a network adapter used when the information collecting apparatus 1 transmits / receives information to / from the Web server apparatuses 2 (# 1) to 2 (#N) (FIG. 1) via the Internet N (FIG. 1). It is. The communication I / F 330 may include a modem, a cable modem, and an Ethernet (registered trademark) adapter.

  The BIOS 350 records a boot program executed by the CPU 310 when the information collecting apparatus 1 is started, a program that the information collecting apparatus 1 depends on hardware, and the like.

  A storage device 410 such as a hard disk 370, an optical disk drive 380, and a semiconductor memory 390 can be connected to the I / O controller 360.

  The hard disk 370 stores various programs for causing the hardware to function as the information collecting apparatus 1, programs for executing the functions of the present invention, the DBs 15 and 17, the information group storage unit 16, and the attribute dictionary 14. The information collecting apparatus 1 can also use a hard disk (not shown) separately provided as an external storage device.

As the optical disk drive 380, for example, a DVD-ROM drive, a CD-ROM drive, a DVD-RAM drive, and a CD-RAM drive can be used. In this case, the optical disk 400 corresponding to each drive is used. A program or data can be read from the optical disk 400 by the optical disk drive 380 and provided to the main memory 340 or the hard disk 370 via the I / O controller 360.

The computer referred to in the present invention refers to an information processing device including a storage device, a control unit, and the like, and the information collection device 1 includes an information processing device including a storage device 410, a control unit 300, and the like. The information processing apparatus is included in the concept of the computer of the present invention .

[Hardware configuration of Web server device]
The Web server device 2 also has a hardware configuration similar to that of the information collection device 1 described above .

[Flowchart According to Embodiment of the Present Invention]
FIG. 5 shows a flowchart of information collection processing according to the embodiment of the present invention.

  S1: The Web document storage unit 11 downloads a Web document distributed on the network N from an arbitrary Web server apparatus 2, and automatically generates the document ID and the communication using the URL of the Web document as an address on the network. The address is associated with and stored in the Web document DB 15. The Web document storage unit 11 stores the document ID of the Web document and the source code that is a description of the Web document in association with each other.

  S2: The information group extraction unit 12 analyzes the source code of the Web document stored in the Web document DB 15, determines the presence of information in the table format or the database format based on the description of the tag included in the source code, When the information in the table format or the database format is found, the information is extracted and stored in the information group storage unit 16.

  As described above, as a method for determining information in a table format or a database format based on a tag, a method for finding a tag constituting a pull-down list, a method for judging an XML declaration, a method for finding a table tag, or the like can be considered.

  In the present embodiment, for example, information posted on the Web page of the PC sales company A shown in FIG. 3B and information posted on the Web page of the PC sales company B shown in FIG. And can be extracted. The information in FIGS. 3B and 3C is assumed to be constituted by table tags as described above.

  S3: The attribute relationship extraction unit 13 analyzes the information in the table format or the database format stored in the information group storage unit 16, and extracts information related to items, attributes, and attribute values included in the information.

At the time of extraction, the attribute relationship extracting unit 13 estimates the location of information having a relationship between items, attributes, and attribute values. Several estimation methods are conceivable.
(1) When the information in the table format or the database format forms a pull-down list by the <Select> tag, for example, the value of the name attribute of the <Select> tag is estimated as an item, and the <Select> tag It is possible to presume that the element contents of the <Option> tag listed in the element contents are attributes and attribute values related to the item. For example,
<Select name = "Note PC">
<Option> CPU xxx 1.5 GHz </ option>
<Option> CPU yyy 2.0GHz </ option>
</ Select>
In the above description, the attribute value “CPU xxx” of the item “notebook PC” and the attribute “CPU” can be estimated. The attribute following the attribute “CPU” can be estimated as “clock”, and the attribute value “1.5 GHz” of the attribute “clock” can be estimated. Similarly, the attribute value “CPU yy” of the item “notebook computer”, the attribute “CPU”, and the attribute value “2.0 GHz” of the attribute “clock” can be estimated.
(2) If the information in the table format or the database format is an XML instance, the upper tag element name forming the hierarchical structure is estimated as “item”, and the tag element name one lower level is set as “attribute” It is possible to estimate, and the attribute or element content of the tag element name indicating the “attribute” can be estimated as the “attribute value”. For example,
<Notebook PC>
<CPU type = “CPU zzz”>
<Clock> 1.1GHz </ Clock>
</ CPU>
<CPU type = “CPU ppp”>
<Clock> 3.2GHz </ Clock>
</ CPU>
</ Notebook PC>
In the above, the attribute value “CPU zzz” of the item “notebook computer”, the attribute “CPU”, and the attribute value “1.1 GHz” of the attribute “clock” can be estimated. Similarly, the attribute value “CPU pp” of the item “notebook computer”, the attribute “CPU”, and the attribute value “3.2 GHz” of the attribute “clock” can be estimated.
(3) When the information in the table format or the database format is configured by a table tag, for example, the table title is estimated as “item”, the element column in the first row is estimated as each “attribute”, It is conceivable that the elements in the second and subsequent rows are estimated as “attribute values” corresponding to the attributes in the same column. For example,
<Table>
<Caption> notebook computer </ caption>
<Tr>
<Td> CPU </ td>
<Td> clock </ td>
</ Tr>
<Tr>
<Td> CPU zzz </ td>
<Td> 1.1 GHz </ td>
</ Tr>
<Tr>
<Td> CPU ppp </ td>
<Td> 3.2 GHz </ td>
</ Tr>
</ Table>
In the above, the attribute value “CPU zzz” of the item “notebook computer”, the attribute “CPU”, and the attribute value “1.1 GHz” of the attribute “clock” can be estimated. Similarly, the attribute value “CPU pp” of the item “notebook computer”, the attribute “CPU”, and the attribute value “3.2 GHz” of the attribute “clock” can be estimated.

  (4) A method using the attribute dictionary 14 is also conceivable. The attribute relationship extraction unit 13 refers to the item registered in the attribute dictionary 14 and determines whether the same item is included in the information A stored in the information group storage unit 16. If the same item is included, the attribute dictionary 14 refers to the attribute associated with the item and determines whether the information A includes the same attribute. If the same attribute is included, the element content immediately after the attribute in the information A is estimated to be the attribute value for the attribute. When the attribute dictionary 14 is used, even if a part of a word that does not require text representing an item or attribute is included, the unnecessary word or phrase can be ignored and the item name or attribute name can be acquired.

  Note that the attribute relationship extraction unit 13 may extract information located immediately above, directly below, or immediately to the left of the tabular information as the item. Further, in the tabular information, the information located in the upper row or the left column may be used as an attribute, and the information located below or to the right may be extracted as an attribute value. Furthermore, the information located immediately to the left of the database format information may be extracted as an attribute, and the database format information may be extracted as the attribute value.

  S4: The attribute relationship extraction unit 13 registers information having the relationship between the item, the attribute, and the attribute value extracted in S3 in the attribute relationship DB 17. In the present embodiment, the attribute relationship extraction unit 13 refers to the attribute dictionary 14 at the time of registration and unifies the item synonyms into one type. For example, the item “notebook PC” obtained from the information shown in FIG. 3B is registered in the attribute relation DB 17 with reference to the item in the attribute dictionary shown in FIG. If it does in this way, it becomes easy to utilize the information of generated attribute relation DB17 by unifying an item. However, a synonym dictionary may be provided separately from the attribute dictionary 14, and the synonyms of items and attributes may be unified by referring to the synonym dictionary. According to the information example shown in FIG. 3, the attribute relationship DB shown in FIG. 3E can be generated from the information in the table format or database format shown in FIGS. 3B and 3C.

  As described above, the information group extracting unit 12 and the attribute relationship extracting unit 13 extract information having a relationship between items, attributes, and attribute values based on tags included in the Web document. Can be automatically collected from a plurality of different Web documents and acquired as information organized by attribute.

  As mentioned above, although embodiment of this invention was described, this invention is not restricted to embodiment mentioned above. For example, “item” is not limited to the example of the present embodiment, and anything with “attribute” and “attribute value” can be “item”. Further, a method for extracting information in a tabular database format and a method for extracting information having a relationship between items, attributes, and attribute values from the information are not limited to the examples in the present embodiment. In addition, each step shown in FIG. 6 does not need to be executed all at once every time one Web document is accumulated. Each step may perform batch processing asynchronously.

  The effects described in the embodiments of the present invention are only the most preferable effects resulting from the present invention, and the effects of the present invention are limited to those described in the embodiments of the present invention. is not.

It is a figure which shows the whole structure of the information collection device which concerns on this embodiment, and a related element. It is a figure which shows the outline | summary of the function structure of the information collection apparatus which concerns on this embodiment. It is a figure which shows the outline | summary of a structure of the web document database which concerns on this embodiment, an information group extraction means, an attribute relation database, and a related element. It is a figure which shows the hardware constitutions of the information collection apparatus which concerns on this embodiment. 3 shows a flowchart of an information collection process according to an embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Information collection apparatus 2 Web server apparatus 11 Web document storage means 12 Information group extraction means 13 Attribute relation extraction means 14 Attribute dictionary 15 Web document DB
16 Information group storage unit 17 Attribute relation DB

Claims (7)

Information gathering device
A first step of extracting information in a table format or a database format from a plurality of Web documents accessible via a communication network based on a tag included in the Web document;
From the extracted information in the table format or database format, based on the dependency relationship between the information indicated by the tag, the relationship between the attribute dependent on the predetermined item and the attribute value indicating the content of the attribute A second step of extracting information comprising:
A third step of associating information having a relationship between the item, attribute, and attribute value for each extracted Web document, and storing the information for each of the associated Web documents as unified information in a storage unit; ,
At least ,
In the information collecting method , the second step extracts, as attributes and attribute values, information that is commonly subordinate to a predetermined item in a number of Web documents exceeding a predetermined threshold .   In the extracting step, the item and the attribute are extracted from the information in the table format or the database format by referring to an attribute dictionary in which the item and the attribute associated with the item are stored. The information collection method according to claim 1, wherein the attribute value is specified from an attribute.   In the storing step, the information extracted from the information in the table format or the database format as the synonym of the item by referring to the attribute dictionary in which the synonym of the item is further stored when storing in the storage unit The information collecting method according to claim 2, wherein items are unified into one type of item.   4. The method according to claim 1, wherein in the extracting step, information located immediately to the left of the database format information is used as an attribute, and the database format information is extracted as an attribute value thereof. 5. Information collection method.   5. The tag according to any one of claims 1 to 4, wherein the tag based on extracting information in the table format or database format is an HTML tag forming a pull-down list, a tag included in an XML document, or an HTML table tag. The information collection method described in the section.   A program for causing a computer to execute the information collecting method according to any one of claims 1 to 5. Information group extracting means for extracting information in a table format or a database format from a plurality of Web documents accessible via a communication network based on tags included in the Web document;
From the extracted information in the table format or database format, based on the dependency relationship between the information indicated by the tag, the relationship between the attribute dependent on the predetermined item and the attribute value indicating the content of the attribute The extracted information is extracted, the information having the relationship between the item, the attribute, and the attribute value is associated with each extracted Web document, and the information for each of the associated Web documents is stored in the storage unit as unified information. Attribute relationship extracting means for
Equipped with a,
The attribute collection device extracts, as attributes and attribute values, information that commonly depends on a predetermined item in a number of Web documents exceeding a predetermined threshold .

Images