KR20000075203A - General-purpose robot agent and real-time search method - Google Patents

Abstract

It is possible to freely operate as an independent system or an integrated system by configuring a template, that is, an independent form, without coding the interconnections of the collectors, indexers, and searchers, which are previously coded in the program, without encoding them. Improves system flexibility by allowing easy integration into other system components

Instead of searching for an index based on a specific term, the indexing method depends on a common format.

The present invention relates to a general-purpose robot agent and a real-time search method, which enables the searcher to search the contents of a website having a quick update cycle by searching a designated website directly instead of searching a database only.

Description

General-purpose robot agent and real-time search method}

Currently serving web search engines (or search sites) are based on a robot agent system, a database, and a search stage, commonly referred to as a robot agent.

Such a conventional web search engine program (or web search system) will be described in detail with reference to FIG. 1 as follows.

The robot agent system 100, the search system (hereinafter referred to as the searcher 200) and the database 300 is generally composed, and the robot agent system 100 is again a collection system (hereinafter collector 10) and It can be divided into an indexing system (hereinafter referred to as indexer 20).

Collector 10 in the robot agent system 100 serves to collect data on a web set in advance, and indexer 20 is generally associated with electronic dictionaries such as noun dictionaries, synonym dictionaries, and dictionaries. It serves as an index for processing.

The indexed data on the web is stored in the database 300 and the searcher 200 searches the stored data.

When users generally search usage information using the web search system, it means that the searcher 200 searches the database 300.

However, such conventional Internet search engines have the following disadvantages.

First, the existing web search system developed for a specific purpose is strongly dependent on the whole program because the collector 10 and the indexer 20 are coded as part of the whole program.

Therefore, the cost is large because the flexibility to expand the function for other purposes or to switch to a system for other purposes, and also requires software knowledge in terms of maintenance.

For example, the above disadvantages will be described.

Web search systems designed for the purpose of searching shopping malls on the web are composed of a collector for collecting only shopping mall websites and an indexer for indexing the data related to the collected shopping malls. Build the program.

However, if the web search system is used for a purpose other than the shopping mall, the collector 10 coded for the shopping mall and the indexer 20 and the searcher 300 coded for the shopping mall in the program are used. ) Must be changed.

This change, in turn, requires significant software knowledge, making free and easy changes difficult.

Conventional indexing methods are through morphological analysis, stopwords, and investigation. However, this process creates difficulties in building an electronic dictionary, and the electronic dictionary itself is dependent on the service application area, and interpretation and processing also inevitably avoid the hard coding in the program, resulting in system-wide flexibility. There is a disadvantage in that scalability is greatly reduced.

The conventional searcher 200 searches for contents in the database 300. Therefore, if the change is not yet reflected in the database 300 after the data on the web, the searcher 200 does not search for the changed content and searches for the existing content.

If the data on the web changes frequently, there is a disadvantage that the existing robot agent cannot retrieve the changed data.

The present invention is to compensate for the drawbacks as described above, without the coding of the collector, the indexer, the searcher, the individual modules that were previously coded in the program template (Template), that is to form an independent form to form an independent system or Improve the flexibility of the system by allowing it to operate freely as an integrated system and easily integrated into components of other systems;

Do not search based on a specific term, but on a common format;

The present invention relates to a general-purpose robot agent and a real-time search method, which enables the searcher to search the contents of a website having a quick update cycle by searching a designated website directly instead of searching a database only.

1 is a schematic diagram of a conventional web search system.

2 is a schematic diagram of a general purpose robot agent and a real-time search method of the present inventors.

The present invention relates to a general purpose robot agent and a real-time retrieval method, all of which are developed in pure Java, and thus can be operated without a separate porting procedure on most existing platforms.

The robot agent 100 is composed of a collector 10 forming a collecting step and an indexer 20 forming an indexing step.

The data indexed through the indexer 20 in the indexing step forms the database 300, and the database 300 is searched through the searcher 200 in the searching step.

When the web search system is driven or initialized, the processor is controlled through an externally provided initialization file. The initialization file is a simple text file that can be easily edited according to the rules, thereby controlling the system.

The initialization file of the collector describes the initial URL of each site, the number of threads, and the collection / exclusion pattern.

The indexer describes site-specific indexing rules, and the searcher describes site-specific collection information / index rules and display formats.

In the program, the control information of the collector 10, the indexer 20, and the searcher 200 is coded, but in the present invention, a template is located to be located outside the system. It can be used as a general purpose robot agent by easily changing the functions located outside the system according to the purpose of use, and each sub system is designed to function as an independent system, which can be freely used for special purposes.

While the existing indexing method is indexed using a specific phrase or word on the web document, the present invention is indexed based on a certain format (constant pattern) shown on the web document. This eliminates the need for a process, which reduces the difficulty of building an electronic dictionary and the electronic dictionary itself, depending on the service application. Eventually system-wide flexibility. Scalability can be greatly improved.

In addition to the existing method of searching the database 300 by the searcher 200 of the search step, the predetermined websites may be directly searched for websites that have not been able to search for the latest updated data by the existing method.

Through this method, it is possible to use this general robot agent and real-time search method for general use such as general search system, shopping mall specialized search system, book information specialized search system.

The operation of the general purpose robot agent having the above characteristics will be described for each module as follows.

Collector 10 is a sub-system of the robot agent and basically performs a web server mirroring function like other existing search systems.

The difference from the existing system is securing flexibility through the template function as described above. For efficient collection control, the collection Skip URL pattern can be specified precisely, and the collection document itself can be limited in association with the index. Also, in consideration of network traffic, parallel processing can be specified for each site.

The indexer 20 is basically controlled by a template, that is, an initialization file.

Unlike conventional indexing systems, no service-dependent electronic dictionaries and program routines are used in the interpretation, parsing, and database processes of documents. Since only the entire document is indexed according to the indexing rules described in the template, it is possible to achieve a high index rate through generality and fast indexing speed.

The indexing rule mechanism starts with the idea that web documents subject to indexing have a certain pattern in information technology.

For example, documents in web shopping malls typically use a unique technology form repeatedly to describe product information (product name, price, manufacturer, etc.) rather than having a separate form for each product. The use of this particular form is the most common phenomenon, especially when serving large amounts of information. After all, if you want to index the information of the shopping mall, the repeated pattern is described as one index rule according to the defined grammar, and the searcher performs the index using this index rule and the document as input.

In addition, if multiple index rules are described on the same site, the searcher automatically selects an index rule that matches the document type. In addition, indexing rules include indexing only documents containing specific words for efficient indexing, processing of extracted data (for example, removing and replacing specific strings), and databaseting of arbitrary information (for example, currency unit or homepage URL). Etc. into an arbitrary field).

The searcher 200 is implemented by Java Servlet technology, and functionally divided into a meta search system and a directory (by topic) search system. As a whole, by applying the template mapping in the same manner as the robot agent 100, the addition of the search site or the change of the user interface component can be performed only by the change of the initialization file. This overcomes the shortcomings of existing browsers, which require the iterative process of modifying, compiling, and deploying programs even with minor information changes.

In addition, one of the biggest obstacles to web applications is stateless (basically the HTTP protocol does not assume Session) and the key to how to search websites with fast update cycles in real time as mentioned above. The following element techniques are used to optimize response time.

The initialization file describes control information for searching, collecting, and indexing a search site most efficiently. Control information includes cache time, retention information type (associated with directory), shortest search path, and index rule (subset of indexer index rule).

In order to minimize the overhead of service through resource management, sparse glass class creation and initialization, thread generation, and shared memory allocation are performed at startup.

Implement an internal cache to take advantage of local databases. The cache consists of a memory cache and a disk cache, and is implemented by keyword (search term) unit and subject (directory) unit. That is, the service request for the same search word and directory is responded with the content of the cache. The Cache Manager executes as a separate thread and automatically performs cache setting, clearing, and switching.

Through the above method, the universal robot agent and the real-time search method can use the search engine used in a specific field in a general search system, a shopping mall specialized search system, a book information specialized search system, and the like.

In addition, the searcher can search fast-updated websites in real time by directly searching a designated website instead of just a database.

Claims (5)

In the web search system for data search on the Internet, A collection step of collecting data of a predetermined website through the collector 10, An indexing step of indexing the data collected in the collecting step through the indexer 20; Converting the data formed through the indexing step into a database (300); General purpose robot agent and real-time search method comprising a search step for searching the database data through a searcher (200). The general robot agent and real-time retrieval method according to claim 1, wherein the subordinate control information of a service is implemented in a template separate from a program in the collection step, the index step, and the retrieval step. The method of claim 1, wherein the indexing of the collected data is based on the description format of the document rather than the content of the document. The general robot agent and real-time search method according to claim 1, wherein the searching step may search a website directly in addition to the method of searching the database data. The general-purpose robot agent and real-time search method according to claim 1, wherein the web search system is entirely developed in Java language.