KR100503148B1 - System for processing web documents based style and content information and method thereof - Google Patents

Abstract

The present invention relates to a web document processing system and method based on style and content information. When the web robot 10 collects web documents according to site information and stores them in the temporary storage 20 (S10), the parser 30 The URL information, structure information, and style information of the web document are parsed and extracted (S20), and the filtering engine 50 performs pattern information and pattern matching on the structure information and style information, and pattern matching is performed. When extracting content included in the lowest node of (S30), the morpheme analyzer 40 extracts nouns from the extracted content (S40), and then the filtering engine 50 checks whether the extracted nouns exist in the unavailable keyword. If it exists in the unused keyword, the web document is regarded as an unused document and the filtering operation is deleted from the temporary storage 20 (S50). If the extracted nouns do not exist in the unused keyword, the corresponding web document Using the structural information and the style information and the available keywords is to extract the necessary information and objects (S60),

Accordingly, it is possible to reduce the time and cost of document management and processing of the system due to the collection of unused documents, it is possible to build a search system with excellent work efficiency.

Description

Web document processing system based on style and content information and method thereof {SYSTEM FOR PROCESSING WEB DOCUMENTS BASED STYLE AND CONTENT INFORMATION AND METHOD THEREOF}

The present invention relates to a web document processing system and a method thereof, and more particularly to the structure of the web document and the unused documents generated in the process of collecting web documents using a web robot, that is, duplicate documents and documents other than information collection targets and The present invention relates to a web document processing system and method based on the style and content information that are removed by using style information and content included in a tag of a web document and extracting necessary information included in the web document.

As is known, the conventional portal site collects web information using a web robot and processes and filters the collected information through a manager determined for each field to provide a web information search service to a user.

However, the conventional method of providing a web information search service to a user in a portal site as described above requires a lot of cost and manpower to manage the collected web information, so that the rapid distribution of web information is not achieved, and the web is duplicated. Although the document can be removed to some extent using a URL, it can hardly perform the deduplication of the web document generated by the CGI used in conjunction with the database, and filtering the web document including the unused information such as the homepage to log in. There is a problem that work is also performed through the administrator.

In particular, in a conventional web document processing system that collects web documents using a web robot, a web document having the same URL is excluded by simply comparing the URLs and processing the duplicate documents by removing the web documents. It is an effective method for web documents existing in a single directory structure to be executed, but it is a problem when web documents are collected from a system that stores a large amount of information in a database and provides web services through CGI and PHP. .

For example, in the case of serving a web document through CGI or the like, there may be a case in which there are a plurality of URLs for one web document, that is, a dynamic parameter is used in CGI, and a web service is performed using CGI. Most homepages use this dynamic CGI.

In addition, processing of documents other than the collection target among the unused documents is performed by a method of directly checking and removing document contents.

Meanwhile, the conventional web document processing system extracts the necessary information (information object) included in the web document by using HTML tags or the structure and style information of the web document, but the wrapper is provided by the institution that provides the web service. There is a problem in that learning time and cost for extracting necessary information by learning the structure and style information of a web document using (Wrapper) are consumed.

Accordingly, the present invention is to overcome the above-described problems, the object of the present invention is to define the structure and style information for the web document to be collected, the use keywords in advance and the structure information and style information of the collected web document, Removes the unused document by matching the nouns contained in the content, selects the structured document style and style information using the structure information and the style information of the web document, and uses the nouns included in the content of the selected lowest node. The present invention provides a web document processing system and method based on style and content information, which are configured to accurately extract information objects included therein.

Web document processing system based on the style and content information for achieving the object of the present invention, Web robot for collecting the web document according to the site information, and storing the collected web document in a temporary storage; A parser for parsing and extracting URL information, structure information, and style information of a web document collected by a web robot and stored in a temporary repository; A morpheme analyzer for extracting nouns from the content included in the lowest node of the structure information of the web document extracted by the parser; And extracting the content by performing the pattern information and the pattern matching operation inputted to the structure information and the style information, and if nouns extracted from the content exist in the unusable keyword, the web document is regarded as unused document and deleted. If it does not exist in an invalid keyword, it consists of a filtering engine that extracts the necessary information objects using structure information, style information, and available keyword.

Web document processing method based on the style and content information for achieving the object of the present invention, collecting the web document according to the site information, and storing the collected web document in a temporary storage; Parsing and extracting URL information, structure information, and style information of a web document collected by a web robot and stored in a temporary repository; Extracting content included in the lowest node of the pattern-matched structure information by performing pattern information and pattern matching on the structure information and style information; Extracting nouns from content included in the lowest node of the extracted web document structure information; Checking whether nouns extracted from the pattern-matched content exist in an unusable keyword and performing a filtering operation to consider the web document as an unusable document and delete it from the temporary storage if it exists in the unusable keyword; And extracting necessary information objects using structure information, style information, and available keyword of the web document if nouns extracted from the pattern-matched content are not present in the unavailable keyword.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1, the web robot 10 collects web documents according to site information and stores the collected web documents in a temporary storage 20.

Parser 30 is collected by the web robot 10 and parses and extracts URL information, structure information, and style information of a web document stored in the temporary storage 20.

The morpheme analyzer 40 extracts nouns from the content included in the lowest node of the structural information of the web document extracted by the parser 30.

The filtering engine 50 extracts the content included in the lowest node of the pattern matched structure information by inputting the pattern information and the pattern matching operation to the structure information and the style information, and inputs it to the stemmer 40. do.

The filtering engine 50 checks whether nouns extracted from the pattern-matched content in the morphological analyzer 40 exist in the unusable keyword. If the unfiltered keyword exists in the unused keyword, the filtering engine 50 considers the web document as an unused document and deletes it from the temporary storage 20. If the filter is not used, the required information object is extracted using the structure information, style information, and available keyword.

The pattern matching module 51 of the filtering engine 50 performs pattern input and pattern matching operations input to the structural information and style information extracted by the parser 30 to include them in the lowest node of the pattern matching structure information. Extracted content and input the extracted content to the morpheme analyzer 40.

The unwanted document filtering module 52 of the filtering engine 50 checks whether nouns extracted from the pattern-matched content in the morphological analyzer 40 exist in the unavailable keyword, and considers the web document as an invalid document if it exists in the unavailable keyword. To perform the filtering operation to delete from the temporary storage (20).

The information object extraction module 53 of the filtering engine 50 extracts the necessary information object using the structure information, the style information, and the available keyword of the web document if nouns extracted as a result of the filtering of the unused document are not present in the unavailable keyword. do.

The schedule manager 54 of the filtering engine 50 drives the pattern matching module 51, the unused document filtering module 52, and the information object extraction module 53 with threads to control input / output of each module. do.

Referring to FIG. 2, the template generator 51a of the pattern matching module 51 generates a template by checking URLs of the collected web documents and receiving pattern information for providing a mapping target structure and style information.

The pattern information inputted to the template generator 51a includes information on processing of an unused document including a URL, a mapping target document structure, and style information included in the document structure, and a URL, a mapping target document structure, and a style included in the document structure. It consists of information object extraction information that consists of information and a range of mapping objects.

The document structure circuit 51b of the pattern matching module 51 extracts nodes having the same structure as the template generated by the template generator 51a while traversing the structure of the web document parsed by the parser 30.

The style mapper 51d of the pattern matching module 51 compares styles included in each node of the document structure extracted by the document structure traverser 51b with style information of the template, and compares each node having the same style. After selection, the web document location information and content of each of the selected nodes are connected to the template and stored in the template storage 51c.

The content extractor 51e of the pattern matching module 51 extracts the content included in the lowest node among the nodes selected by the style mapper 51d in text form and inputs the text to the stemmer 40.

Referring to FIG. 3, the invalid keyword checker 52a of the invalid document filtering module 52 checks whether nouns extracted from the morpheme analyzer 40 exist in the invalid keyword.

The unused document remover 52b of the unused document filtering module 52 determines that the collected web document is an unused document when the extracted nouns exist in the unused keyword as a result of the unused keyword inspection, and collects the web document collected in the temporary storage 20. And delete the template stored in the template storage 51c. At this time, the web document deleted by the unnecessary document remover 52b is a document other than the information collection target.

The duplicate document remover 52c of the unused document filtering module 52 performs a redundancy check of previously collected web documents and currently collected web documents to remove duplicate documents from the template storage 51c. Templates having the same content by comparing the contents of the currently input templates with the same contents are removed as duplicates, and the collected web document and the template stored in the template storage 51c are removed.

Referring to FIG. 4, the information object inspector 53a of the information object extraction module 53 receives the information object extraction information from the template storage 51c and inspects nodes having document structure and style information for extracting the information object. After that, the content existing at the node end is extracted and input to the morpheme analyzer 40.

The information object selector 53b of the information object extraction module 53 checks whether the extracted nouns exist in the available keyword when the nouns are extracted from the content extracted by the morpheme analyzer 40, and the hit rate of the test result. By calculating the hit rate, the content is extracted to the required information object if it exceeds a certain threshold.

The web document processing system based on the style and content information according to the present invention configured as described above operates as follows by the method as shown in FIG.

Initially, the web robot 10 collects web documents according to site information, stores the collected web documents in the temporary storage 20, and inputs the collected web documents into the parser 30 (S10). .

In addition, when the collected web document is input to the parser 30, the parser 30 parses and extracts URL information, structure information, and style information of the web document input after being collected by the web robot 10. (S20).

After the URL information, the structure information, and the style information of the collected web document are extracted, the pattern matching module 51 of the filtering engine 50 first extracts the pattern information and the style information from the extracted structure information and style information. After the pattern matching operation is performed to extract the content included in the lowest node of the pattern-matched structure information, the extracted content is input to the morpheme analyzer 40 (S30).

In addition, when the extracted content is input to the morpheme analyzer 40, the morpheme analyzer 40 extracts nouns from the content included in the lowest node of the extracted structure of the web document and inputs the nouns to the filtering engine 50 ( S40).

Next, the dead document filtering module 52 of the filtering engine 50 checks whether nouns extracted from the pattern-matched content exist in the unusable keyword, and if the extracted nouns exist in the unused keyword, converts the web document into an unused document. It is considered to perform the filtering operation to delete from the temporary storage 20 (S50).

In this case, the extracted document filtering module 52 determines that the collected web document is an unused document, that is, a document other than the information collection target, when the extracted nouns exist in the unavailable keyword and the web document collected in the temporary storage 20. Templates previously stored in the template storage 51c to perform an operation for removing an unused document to delete the template stored in the template storage 51c, and to perform a duplicate inspection of the previously collected web documents and the currently collected web documents. By comparing the contents of the currently input templates, templates having the same content are recognized as duplicates, and a duplicate document removal operation of removing the stored web documents and the templates stored in the template storage 51c is performed together.

On the other hand, as the result of checking whether nouns extracted from the pattern-matched content exist in the unusable keyword, the unused document filtering module 52 of the filtering engine 50 finds that the information object extracting module ( 53) the required information object is extracted using the structure information, the style information, and the available keyword of the web document (S60).

In this case, the information object extraction module 53 of the filtering engine 50 receives the information object extraction information from the template repository, first checks the nodes having the document structure and style information to extract the information object, and then at the node end. After the existing content is extracted in the form of text, the extracted content is input to the morpheme analyzer 40 so that the morpheme analyzer 40 extracts nouns from the extracted content.

When nouns are extracted from the morpheme analyzer 40, the information object extraction module 53 checks whether the extracted nouns exist in the available keyword, and then calculates and sets a hit rate of the test result in advance. If the content is above a certain threshold, the content is extracted as the required information object so that the required information object is searched through the search engine.

As described above, the web document processing system and method based on the style and content information according to the present invention are used to generate an unused document generated in the process of collecting a web document using a web robot. The contents contained in the tag are removed in the web document collection step of the web robot, and the information objects contained in the web document are accurately extracted. Therefore, the management and maintenance of the documents collected during the construction and operation of the portal site, etc. In addition to optimizing and reducing the cost and manpower required to operate the site, it also has the advantage of minimizing the waste of network resources of the web robot.

In particular, since unused documents can be solved systematically, the system can not only reduce the time and cost of document management and document processing due to the collection of unused documents, but also collect only the necessary documents, thereby providing a highly efficient search system. It is possible to build, and it is advantageous to easily and smoothly build and operate services such as the latest news article service and searching for a specific product in a web shopping mall.

What has been described above is only one embodiment for implementing the web document processing system and method based on the style and content information according to the present invention, the present invention is not limited to the above-described embodiment, the following claims Without departing from the gist of the present invention claimed in the scope, anyone of ordinary skill in the art will have the technical spirit of the present invention to the extent that various modifications can be made.

1 is a block diagram showing a web document processing system based on style and content information according to the present invention;

2 is a block diagram showing a pattern matching module,

3 is a block diagram illustrating an unused document filtering module;

4 is a block diagram showing an information object extraction module;

5 is a flowchart illustrating a web document processing method based on style and content information according to the present invention.

<Description of the symbols for the main parts of the drawings>

10: Web Robot 20: Temporary Storage

30: Parser 40: Morphological Analyzer

50: filtering engine 51: pattern matching module

52: filtering of the unused document 53: information object extraction module

Claims (9)

A web robot that collects web documents according to site information and stores the collected web documents in a temporary storage; A parser for parsing and extracting URL information, structure information, and style information of a web document collected by a web robot and stored in a temporary repository; A morpheme analyzer for extracting nouns from the content included in the lowest node of the structure information of the web document extracted by the parser; And Extracts content by performing pattern matching and pattern matching on the structure information and style information, and if nouns extracted from the content exist in an unusable keyword, the corresponding web document is regarded as an unused document, deleted, and unused. If the keyword does not exist, a filtering engine is provided to extract necessary information objects using structure information, style information, and available keyword keywords, and the filtering engine includes pattern information inputted to the structure information and style information extracted from the parser. A pattern matching module for extracting content included in the lowest node of the pattern-matched structure information by performing a pattern matching operation and inputting the content to the morpheme analyzer; An unused document filtering module for checking whether nouns extracted from the pattern-matched content exist in the unusable keyword, and if the unused keyword exists in the unused keyword, considers the web document as an unused document and deletes it from the temporary storage; An information object extraction module for extracting necessary information objects using structure information, style information, and available keyword of the web document if nouns extracted as a result of filtering the unwanted document are not present in the unavailable keyword; And a schedule manager for driving the pattern matching module, the unused document filtering module, and the information object extraction module with threads, respectively, and controlling the input / output of the respective modules. delete The method of claim 1, wherein the pattern matching module A template generator for checking a collected web document URL and receiving a pattern information for providing a mapping target structure and style information to generate a template; A document structure traversal unit which traverses the structure of the web document parsed by the parser and extracts nodes having the same structure as the template generated by the template generator; After comparing the styles included in each node of the document structure extracted by the document structure traversal with the style information of the template, the nodes having the same style are selected, and the web document position information and content of each of the selected nodes are selected. A style mapper that associates a template with a template and stores the template in a template storage; And A content extractor for extracting content contained in the lowest node among the nodes selected by the style mapper in a text form and inputting the content to the stemmer Web document processing system based on the style and content information, characterized in that consisting of. The method of claim 3, wherein the pattern information input to the template generator is Unused document processing information consisting of a URL, a mapping target document structure, and style information included in the document structure; And Information object extraction information composed of URL, mapping target document structure, style information included in the document structure, and range of mapping target structure Web document processing system based on the style and content information, characterized in that consisting of. The method of claim 1 or 3, wherein the unnecessary document filtering module An invalid keyword checker for checking whether nouns extracted from the morpheme analyzer exist in the invalid keyword; An unused document eliminator for deleting the web documents collected in the temporary storage and the templates stored in the temporary storage by determining the collected web documents as unusable documents when the extracted nouns exist in the unusable keywords; And In order to perform a duplicate check of previously collected web documents and currently collected web documents, templates previously stored in the template storage and contents of the currently input templates are compared, and templates having the same content are recognized as duplicates. Duplicate document remover to remove documents and templates stored in template storage Web document processing system based on the style and content information, characterized in that consisting of. The method of claim 1, wherein the information object extraction module An information object inspector that receives the information object extraction information from the template repository, inspects nodes having document structure and style information for extracting the information object, and extracts the content existing at the end of the node and inputs it to the morpheme analyzer; And When nouns are extracted from the content extracted by the morpheme parser, the extracted nouns are examined in the available keyword, and the hit rate of the test result is calculated to extract the content as the required information object if it is above a certain threshold. Information object selector Web document processing system based on the style and content information, characterized in that consisting of. Collecting web documents according to site information and storing the collected web documents in a temporary storage; Parsing and extracting URL information, structure information, and style information of a web document collected by a web robot and stored in a temporary repository; Extracting content included in the lowest node of the pattern-matched structure information by performing pattern information and pattern matching on the structure information and style information; Extracting nouns from content included in the lowest node of the extracted web document structure information; Checking whether nouns extracted from the pattern-matched content exist in an unusable keyword and performing filtering operation to consider the web document as an unused document and to delete it from the temporary storage if it exists in the unusable keyword, wherein the extracted nouns Removing the unused document from the web document and the template stored in the temporary storage by determining the collected web document as an unused document when the unused keyword exists; And comparing templates previously stored in the template storage and contents of the currently input templates to perform duplicate inspection of the previously collected web documents and the currently collected web documents. Performing a filtering operation including a duplicate document removing step of removing a template stored in a web document and a template storage; And Extracting necessary information objects using structure information, style information, and available keyword of the web document if nouns extracted from the pattern-matched content are not present in the unavailable keyword. Web document processing method based on the style and content information, characterized in that consisting of. delete 8. The method of claim 7, wherein extracting the required information object Inspecting nodes having document structure and style information for extracting the information object and extracting content existing at the end of the node; Extracting nouns from the extracted content; And Checking whether the extracted nouns exist in the available keyword, and calculating the hit rate of the test result and extracting the content as the required information object if it exceeds a certain threshold. Web document processing method based on the style and content information, characterized in that consisting of.