JP2003303200A - Knowledge information collecting system and knowledge information collecting method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently collect as knowledge information various types of document files having different file formats from the Internet. <P>SOLUTION: When a setting file 203 describes setting information 'collect only the inside of a domain', a Web collecting module 111 collects document information only inside the domain of a set origin URL from a network based on the set contents on a collection site setting screen. Thus, if necessary, knowledge information can be efficiently collected with the collection site narrowed down to the inside of the domain, and common knowledge information can be easily, rapidly, and efficiently collected in a limited network range. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a knowledge information collecting system and a knowledge information collecting method used in a knowledge management system, and particularly to a knowledge information collecting system and knowledge for collecting document information to be registered in a knowledge database from a network. Regarding information collection method.

[0002]

2. Description of the Related Art In recent years, the introduction of groupware for sharing information among a plurality of users has been promoted mainly in companies. E-mail systems and workflow systems are known as typical groupware,
Recently, a knowledge management system to support sharing of knowledge information is also being developed.

This knowledge management system is for accumulating and managing knowledge information such as personal know-how in a knowledge database. By combining it with a search function such as natural language search, the accumulated knowledge information can be efficiently stored. It is possible to make full use of it.

By the way, in such a knowledge management system, how to efficiently collect and store knowledge information is an important point. In particular, since Web information on the Internet is a treasure trove of knowledge, a mechanism for efficiently collecting necessary information from the Internet is needed.

[0005]

Recently, however, not only text-based document files such as HTML documents but also document files of various file formats created by various application programs have begun to be used for homepages. In this case, if the document file acquired from the Internet is directly registered in the knowledge database, it becomes difficult to search the document file in the knowledge database, and the knowledge cannot be effectively used.

The present invention has been made in consideration of such circumstances, and a knowledge information collecting system and a knowledge information collecting system capable of efficiently collecting various types of document files having different file formats from the Internet as knowledge information. The purpose is to provide a method for collecting knowledge information.

[0007]

In order to solve the above-mentioned problems, the present invention is a knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, which should be registered in the knowledge database. Setting screen display means for displaying a collection site setting screen including a setting field of a starting point URL for setting a site for collecting document information and an item for defining a collection range, and a setting displayed by the setting screen display means A setting file creation means that creates a setting file based on the settings on the screen,
And an information collecting means for collecting from the network the document information to be registered in the knowledge database based on the contents of the setting file created by the setting file creating means. And a plurality of types of setting items including setting items for which the collection range is within the domain, and the information collecting means sets the starting point URL and the content for which the collection range is within the domain in the setting file. In this case, the document information is collected from the network according to the setting contents of the collection site setting screen, only within the domain of the set origin URL.

In this knowledge information collecting system, the information collecting means executes the collection of the document file from the network based on the setting information designating the collecting condition of the information from the network. When the starting point URL and the content of which the collection range is within the domain are set in the setting file, the information collecting means sets the collecting site setting screen for the collecting site setting screen only within the domain of the set starting point URL. The document information is collected from the network according to the setting contents. This allows knowledge information to be efficiently collected by narrowing down the sites to be collected within the domain, and shared knowledge information can be easily and quickly collected efficiently within a limited network range. .

Further, according to the present invention, in a knowledge information collecting method for collecting document information to be registered in a knowledge database from a network, a site for collecting document information to be registered in the knowledge database is set. A setting file is displayed based on the setting screen display step for displaying the collection site setting screen including the URL setting field and the item defining the collection range, and the setting contents on the setting screen displayed by the setting screen display step. The method includes a setting file creating step to be created, and a collecting step of collecting document information to be registered in the knowledge database from a network based on the contents of the setting file created in the setting file creating step. In the setting screen, the setting field of the origin URL and the collection range are within the domain Together comprising a constant item, U for collecting
At least one of the items to set the RL pattern, the items to set the pattern of URLs that are not collected, the items to set the file format to be collected, the items to set the upper limit of the links to be collected, and the items to set the upper limit of the number of files to be collected. In the configuration file, the collecting step includes
When the origin URL and the content having the collection range in the domain are set, the document information is acquired from the network according to the setting content of the collection site setting screen, targeting only the domain of the set origin URL. It is characterized by collecting.

In this knowledge information collecting method, the information collecting means executes the collection of the document file from the network based on the setting information designating the collecting condition of the information from the network. In the collecting step, when the origin URL and the content having the collection range in the domain are set in the setting file, the collection site setting screen is set only in the domain of the set origin URL. According to the contents, the document information is collected from the network. This will allow you to
Knowledge information can be efficiently collected by narrowing down the sites to be collected within a domain, and shared knowledge information can be easily and quickly collected efficiently within a limited network range. Furthermore, in addition to setting the collection range to be within the domain, other collection conditions can be combined and collected, so that desired knowledge information can be more appropriately narrowed down and collected, whereby a higher-level knowledge database can be collected. Can be constructed easily, quickly and efficiently.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of a knowledge management system using a knowledge information collecting system according to an embodiment of the present invention. This knowledge management system is for providing services such as collection, analysis, and search of knowledge information, and includes a Web collection system 11, a registration module 12, and a knowledge engine 13.
Etc. These Web collection system 1
1, the registration module 12, and the knowledge engine 13 are realized as a program executed on the server computer. The web collection system 11 and the registration module 12 constitute a knowledge information system for collecting knowledge information used in the knowledge management system. This knowledge information collecting system collects documents such as Web pages from the Internet / Intranet 30 and registers the contents of the documents in the knowledge database (knowledge DB) 131 of the knowledge management system. Web collection system 11
Collection module 111, management interface 112,
And a registration directory 113.

The Web collection module 111 is a program for collecting various document files on the Internet / Intranet 30 and outputting them in a format that can be registered in the knowledge DB 131. This Web collection module 111 has a multi-thread structure, and HTTP
Document file acquisition processing from the Internet / Intranet 30 by (HyperText Transfer Protocol) can be performed in parallel. HTTP is an information site on the Internet / Intranet 30 for each Web
From server 31 to HTML (Hyper Text Markup Languag
It is a communication protocol for acquiring the document described in e). The web collection module 111 uses the URL (Uniform Reload) specified by the setting information that specifies the web collection conditions.
Source Locator) document file is acquired from the Internet / Intranet 30, and if the acquired document file contains the URL of the link destination, the document file of that URL is further acquired. , The related document files are sequentially acquired from the Internet / Intranet 30. In this case, it is necessary to analyze the acquired document file in order to detect the link information (URL) to another document file. In the analysis process, not only the detection of the link information but also the knowledge Extraction of text data to be registered in the DB 131 is also executed.

The document file to be collected is created not only by a hypertext file (HTML file) capable of describing a URL that is link information to another file, but also by a text file (plane text) and various application programs. Document files in various file formats (for example, Portable Docu
ment Format files, document files created with word processor software, spreadsheet files, presentation data files created with presentation data creation software, etc.) Become.

The Web collection module 111 acquires attribute information to be registered in the knowledge DB 131 and extracts the above-mentioned text data for each document file collected from the Internet / Intranet 30. The attribute information acquired from each document file is, for example, the URL of the document file, file creation date / time (update date / time), and the like. Then, the Web collection module 111 creates a registration file in which the attribute information of each of the document files and the text data are described, and outputs it to the registration directory 113. Here, the registration file is
It means a file having a predetermined description format as a common interface for registering document information of each of a plurality of types of document files having different file formats in the knowledge DB 131. This registration file is used to register the document information of each of a plurality of types of document files in the knowledge DB 131 in a common format. In this embodiment,
XML (eXtensible Markup Langua) as a registration file
ge) is used.

The management interface 112 is a program for setting the contents of each Web collection process (hereinafter referred to as a Web collection process task) to be executed by the Web collection module 111, and the Web collection conditions for each Web collection process task. Of the Web collection module 111, and the functions of managing and presenting the collection status of each Web collection processing task. This management interface 112 allows the web server 22 to execute CGI (Common) so that an administrator user can perform necessary operations on the web browser 21 of the user terminal.
It is realized as an external program (CGI program) that can be started through the Gateway Interface.

Further, the Web collection system 11 includes a Web
As a file for managing and controlling the operation of the collection module 111, a lock file 201,
A control file 202, a setting file 203, a result file 204, and a log file 205 are provided.

The lock file 201 is a file for exclusive control for preventing double activation of the Web collection module 111. The control file 202 is a file used by the management interface 112 to stop the Web collection module 111. For example, in the case of interrupting a Web collection processing task that is being executed in response to an instruction from an administrator user. Used for. The administrator user can specify the interruption during the collection execution through the management interface 112.

The setting file 203 is a file for describing the setting information for designating the Web collection condition for each Web collection processing task, and here, a plurality of setting information corresponding to each of the plurality of Web collection processing tasks is set. Can be held. The web collection condition of each web collection processing task is set by the administrator user. In the Web collection condition, at least the origin address information (origin URL) indicating the location of the document file to be the origin of the information collection from the Internet / Intranet 30 and the collection range are specified only within the domain, or the URL for collection is specified. Includes multiple types of collection range information, such as specifying the pattern of URLs that are not collected, specifying the maximum number of document files or links that should be collected, and specifying the file format to be collected. Has been. This Web
We of the Web collection module 111 based on the collection condition
b The collection operation is controlled. That is, the Web collection module 111 sequentially collects the document file group related to the document file specified by the starting point URL within the range defined by the collection range information. The plurality of types of collection range information described in the setting file 203 and the setting means will be described later with reference to the GUI screen.

The Web collection condition also includes a knowledge database name that specifies the knowledge database of the registration destination. That is, in the knowledge DB 131, a plurality of knowledge databases having different types of accumulated documents are defined, and the plurality of knowledge databases are managed by respective knowledge database names. By designating the knowledge database name of the registration destination for each Web collection task by the setting information of the setting file 203 described above, the collected document information can be registered in any knowledge database in the knowledge DB 131.

Further, as the collection range information, a collection condition for recollecting can be set. Here, recollection means, for example, the same W as the Web collection module 111.
The eb collection processing is the second and subsequent Web collection processing when the eb collection processing task is periodically executed, and the Web collection processing when the once interrupted Web collection processing task is restarted.

The result file 204 is a file for managing a list of document files collected as a result of Web collection processing for each Web collection task, and here, a list of URLs acquired as a result of Web collection. ,
The number of acquired document files and the like are output for each Web collection processing task. This result file 204 is a web
It is used for the purpose of presenting the collection status of each collection processing task to the user and for the purpose of improving the efficiency of recollection. By using this result file 204, the Internet / Intranet 3 can be used among the document files collected in the past.
It is also possible to detect a document file deleted from the information site on 0 or a restart point when the Web collection process is interrupted midway.

The log fill 205 is a file for managing the success or failure of acquisition of a document file by the Web collection module 111 and the type of error.

The registration directory 113 is a knowledge DB 11
3 is a storage area for outputting the content of the document to be registered. The registration file 114 in which the attribute information and the text data extracted from each document file by the Web collection module 111 are described is output here.
The web collection module 111 has two processing modes, first and second, regarding extraction of text data.

In the first processing mode, analysis processing is performed on document files of all file formats, text data is extracted, and link information (URL) is detected (URL is detected only in HTML files. Target).
In the second processing mode, only the HTML file and the text file (PlainText) are analyzed to extract the text data and further the link information (URL) is detected (URL is detected only in the HTML file. ), For example, Portable Document F
No analysis processing is performed on the file in the ormat format.

When the second processing mode is used, HTML
For PlainText and PlainText, the text data and attribute information are described in the registration file 114 and output to the registration directory 113. On the other hand, Portab
For files of other formats such as le Document Format files, the content file 115 is output as it is to the registration directory 113, and the registration file 114 describes the attribute information of the file and the path name of the content file 115. .
Note that in the second processing mode, it is important that the text data extraction is also executed in the analysis process for the file that requires the analysis process for detecting the link information. The text data may be extracted only for the HTML file.

In both of the first and second processing modes, the Web collection module 111 executes the Web being executed.
When the processing of the collection task is completed, the registration module 12
A registration request file is issued to request registration of the document from the registration module 12. In the registration request file,
The file name of the registration file and the knowledge database name of the registration destination are described.

The registration module 12 registers the attribute information and text data of each document file collected by the Web collection module 111 in the knowledge DB 131.
That is, the registration module 12 acquires the corresponding registration file 114 in response to the registration request from the Web collection module 111, and extracts the attribute information and text data described for each document file on the registration file 114. , The registration destination knowledge database of the knowledge DB 131 is registered. The registration module 12 also has a text data extraction function, and the registration file 11
When the path name of the content file 115 is described in 4, the registration module 12 extracts text data from the content file 115 specified by the path name and registers it in the corresponding registration destination knowledge database of the knowledge DB 131. .

Further, the registration module 12 also executes a registration process of the documents collected by the collection modules other than the Web collection module 111. Other collection modules include, for example, a file collection module that collects document information from a file server of an electronic filing system, an RDB collection module that collects document information managed by the RDB database server, and an electronic bulletin board format community. There are a community collection module that collects the document information posted therein, a user collection module that the user uses to convert the document files of various application programs into the output format of the registration file (XML), and the like. The registration module 12 is commonly used by each of these collection modules. Registration module 1 for all collection modules
The above-mentioned XML-formatted registration file is used as an interface for 2.

The knowledge engine 13 performs a knowledge analysis process for utilizing the information accumulated in the knowledge DB 131. In this knowledge analysis process, a process of analyzing the features of each of a large amount of document information accumulated in the knowledge DB 131 to extract important words, a clustering process of classifying and organizing features into a plurality of categories, and the like are performed. This knowledge engine 13 includes a knowledge engine 132 for performing a natural language search.
Is provided for each user, and each user
By accessing the knowledge engine 132 from the browser 41 through the web server 42, the web browser 4
The knowledge accumulated in the knowledge DB 131 can be searched from above. Since the attribute information of each document stored in the knowledge DB 131 also includes the URL of the document, the original document is acquired from the corresponding information site and we
b It can be displayed on the browser 41. The search process can be performed by selecting a certain knowledge database name and targeting only the selected knowledge database, or by targeting all knowledge databases in the knowledge DB 131.

Next, the functional configuration of the Web collection module 111 will be described with reference to FIG. The web collection module 111 includes a collection control unit 301,
It has an attribute extraction unit 302, a text extraction unit 303, and a format conversion unit 304. Collection control unit 301
For each of all the setting information held in the setting file 203, the Web specified by the setting information
Run collection tasks one at a time. A group of related document files is sequentially collected from the Internet / Intranet 30 starting from the starting point URL designated by the setting information for each Web collecting task. The URL to the linked document included in the acquired document file is the URL list 30
5, the collection processing is recursively executed while acquiring URLs from the URL list 305. URL
The result file 204 described above can be used as the list 305. The collection range is limited by the collection conditions held in the setting file 203.

Each collected document file is first sent to the attribute extraction unit 302, where the attribute information of each document file is acquired. Next, the text extraction unit 303 analyzes the document and extracts the text data to be registered in the knowledge DB 131 and the link destination URL to be acquired next. For example, for an HTML file, extraction of text data is performed by extracting a portion other than the HTML tag. The extracted text is divided into a title and a body (body). To obtain the link destination URL, the A tag, the HREF value of the AREF tag, FRAME
This is performed by acquiring the SRC value of the tag, the IFRAME tag, the LAYER tag, and the REFRESH value of the META tag. When operating in the second processing mode described above, the processing of the text extraction unit 303 is performed only on the HTML file and the text file, and Port
It is not performed for a document file of another file format such as a file of the Document Format format.

The attribute information and text data acquired from each document file are sent to the format conversion unit 304,
Then, it is formatted into an output format according to the description format of the XML registration file and is output on the registration directory 113. In one registration file 114, for example, 100
Attribute information and text data about 0 document files are described. The attribute information and path name of the document file for which text extraction has not been performed are described in the registration file 114.

Next, an example of the output format of the registration file 114 will be described with reference to FIGS. 3 and 4.

FIG. 3 shows an example of an output format when the text extraction unit 303 extracts text. Tag at the beginning of the file <? Xml version = "1.0" encod
ing = "Shift_JIS"?>, <KnowledgeSystem> indicates the beginning of the file, and the end tag </ KnowledgeSystem>
Indicates the end of the file.

Attribute information and text data of one document file is described in each record enclosed by <RECORD> and </ RECORD>. The meaning of the tag in each record is as follows.

MODE: Mode information for designating the operation mode of the registration module 12. This mode information instructs registration or deletion of document information (text data and attributes) in the knowledge DB 131 for each document file.
Either 2 = registration (overwrite) or 0 = delete. In case of deletion, only TYPE and UNIQUE of the following tags are output.

TYPE: Indicates the type of collection. In this example, it is always "Web collection". UNIQUE: A unique key for identifying the document file registered in the knowledge DB 131. Usually, the URL of the document is used as a unique key.

ORGDATE: Indicates the creation date (or update date) of the document file. TITLE: Indicates the title of the document file. The text data extracted from the title part of the HTML file becomes the title. No title is output for files other than HTML files. This title is output as the title of the document file on the search screen.

AUTHOR: Describe the host name (host address of URL) of the information site that owns the document file. DATE: Describe the date part of ORGDATE above. URL: URL of the document file. It has the same value as UNIQUE. BODY: The text data extracted from the document file is described.

FIG. 4 shows an example of an output format relating to a document file in which the text extraction unit 303 has not performed text extraction.

No text data is described in BODY.
In PATH1 in the area enclosed by BDYFILE></BDYFILE>, the path name to the content file 115 output on the registration directory 113 is described. DEL = 1 instructs the registration module 12 to delete the original file in the registration directory 113 after extracting the text data from the content file 115. We
When the b collection module 111 operates in the second processing mode, text data is described in BODY for HTML files and plain text, and BD for document files (content files) in other file formats.
The path name of the content file will be described in YFILE.

Next, referring to the flowchart of FIG.
Web collection module 1 for collected document files
A procedure of a series of processing executed in 11 will be described.

First, from the collected document files, the knowledge DB
Attribute information to be registered in 131 (URL, AUTHOR, ORGDATE,
DATE) is acquired (step S101). The attribute information can be acquired by using the value returned from the information site by HTTP, the value added in the collected document file, or the like. After that, in the second processing mode, the file type is determined based on the extension of the collected document file, and the file is an HTML file or a plain text file, or a file of other file format. Is determined (steps S102 and S103). The collected document file is H
If the file is a TML file or a plain text file (YES in step S103), the above-mentioned text extraction processing (text extraction and link URL detection in the case of an HTML file) is executed (step S10).
4) Then, the attribute information and the text data are described in the above-mentioned format on the registration file 114 (the text data is inserted into BODY) (step S105). On the other hand, if the file has a file format other than the HTML file or the plain text file (NO in step S103), the file is the registration directory 1
13 is output as it is (step S106), and then the attribute information and the path name of the file are registered file 1
The process described in 14 (the path name is described in BDYFILE) is performed (step S107).

If the document file of the target URL cannot be acquired from the Internet / Intranet 30, the content of the document file is already in the knowledge DB1.
Mode information (MOD
E) = 0 (delete) is described on the registration file 114.

Further, in the first processing mode, the processes of steps S102 and S103 are not performed, and the processes of steps S104 and S105 are executed for all the acquired files.

Next, referring to the flowchart of FIG.
The procedure of the registration process executed by the registration module 12 will be described.

The registration module 12, which has received the registration request from the Web collection module 111, retrieves the records one by one from the registration file (XML file) specified by the registration request file from the Web collection module 111, and Process. First, is the mode information in the processing target record MODE = 0 or MODE = 2
Is checked (steps S111 and S11).
2).

When MODE = 2, the registration module 12
Is the data item (TYPE, UNIQUE, ORGDATE, TITLE, AUTH) described in each tag according to each tag in the record.
The contents of OR, DATE, URL, and BODY) are registered in the registration destination knowledge database in the knowledge DB 131 specified by the registration request file (step S113). If the text data does not exist in the BODY tag, the text data will not be registered. Then, it is judged whether or not the path name is described in the tag of BDYFILE (step S114). If the path name is described (YES in step S114), the corresponding content file is acquired from the storage area specified by the path name (step S115), and the process of extracting text data from the content file is executed. Performed (step S1
16). Then, the extracted contents are registered in the registration destination knowledge database as text data of the corresponding document file (step S117).

When MODE = 0, the attribute information and text data of the registered document file designated by UNIQUE in the record are searched from the knowledge DB 131, and the registered content is deleted from the registration destination knowledge database (step S118).

FIG. 7 shows the relationship between the state of the document file (Web content) on the Internet / Intranet 30 and the registration / deletion process to be performed on the knowledge DB 131. In the present embodiment, the following processing is performed based on the policy of reflecting the state of the latest Web content on the Internet / Intranet 30 in the knowledge DB 131 as much as possible.

(1) The updated Web content is overwritten on the knowledge DB 131 (MODE = 2). Updated W
When registering the attribute information and text data of the eb content, the UNIQUE is the same as the UNIQUE of the pre-update Web content already registered in the knowledge DB 131. Therefore, MODE = 2 updates (overwrites) the attribute information and text data of the pre-update Web content already registered in the knowledge DB 131 with the updated Web content attribute information and text data.

(2) The added Web contents are additionally registered in the knowledge DB 131 (MODE = 2). When registering the attribute information and text data of the added Web content, the UNIQUE is not registered in the knowledge DB 131. Therefore, by setting MODE = 2, the attribute information and text data of the added Web contents are stored in the knowledge DB 13.
1 can be additionally registered.

(3) The deleted Web contents are also deleted from the knowledge DB 131 (MODE = 0).

Next, the functions provided by the management interface 112 will be specifically described. As described above, the management interface 112 sets the Web collection task settings /
It is a user interface for managing and executing,
1) A function of creating a plurality of settings of a web collection task and saving them in the setting file 204. 2) A list of collection settings,
The administrator user is provided with a function of performing management work such as deletion, 3) a function of starting and ending (suspending and resuming) collection, and 4) a function of confirming the status and achievement of collection in real time.

The screen displayed on the Web browser 21 of the administrator user by the management interface 112 is composed of three screens of "list screen", "setting screen" and "status screen" as shown in FIG. When the management interface 112 is activated, the list screen is first displayed.
The three screens transit according to the operation performed by the administrator user on each screen. The list screen displays a list of currently created settings and the latest collection result of each setting. On this list screen, for example, when a created setting is selected and the selected setting transitions to the setting screen or status screen related to the setting, addition / deletion of the setting content can be performed on the setting screen. It is possible to perform, and on the status screen, it is possible to specify activation and termination (suspension and resumption) of the setting. If a new setting is selected on the list screen, the setting screen is displayed, and a process for creating a new setting can be performed on the setting screen. The list screen, the setting screen, and the status screen will be specifically described below.

FIG. 9 shows an example of the list screen.
On this list screen, as shown in the figure, "List" anchor 5
01, “Setting anchor” 502, and “Status” anchor 503 are displayed. These “list” anchor 501, “setting anchor” 502, and “status” anchor 503 are links to the list screen, setting screen, and status screen, respectively. When the “list” anchor 501 is clicked on the list screen, the display content of the list screen is updated to the latest state. Further, on the list screen, the "new setting" field 50
4, "Create" button 505, and status list display section 50
6 is provided. Enter the setting name of the Web collection task to be newly created in the "New setting" field 504,
When the "Create" button 505 is clicked, it is possible to transition to the setting screen and newly create setting information regarding the setting name.

The status list display unit 506 displays a list of collection statuses for each of the Web collection tasks that have already been created and registered, based on the contents held in the result file 204 described above. The following items are displayed for each Web collection task.

"Setting name": Name of registered setting "Knowledge database name": Name of database to register collected data "Status": Current status of setting (uncollected / collecting / finished / interrupted / abnormal) Either end) "Start": Date and time of last start "End": Date and time of last end "File": Number of files collected by the previous task execution
(This time) and the maximum value of the total number of files collected so far (whole) As described above, in this embodiment, a plurality of Web collection tasks can be set and Multiple We
b The collection status of each collection task can be confirmed on the list screen.

Next, an example of the setting screen will be described with reference to FIG. On the setting screen, as shown in the figure, in addition to the “list” anchor 501, the “setting anchor” 502, the “status” anchor 503, a “setting name” field 601,
“Knowledge database name” field 602, “proxy server” setting form 603, “collection file format” setting form 604, “acquisition request” setting form 605,
An “authentication information” setting form 606 and a “collection site” setting form 607 that constitutes a collection site setting screen are provided, and the following setting editing operations can be performed using these.

(1) Setting name: "Setting name" field 6
In 01, the setting name designated by creating a new setting or selecting an existing setting on the list screen is displayed. (2) DB selection: By inputting the knowledge database name in the “knowledge database name” field 602, the registration destination knowledge database for registering the collected data is selected.

(3) Proxy: The proxy server used for collection is set using the "proxy server" setting form 603. (4) File format to collect: “Collecting file format”
A file format to be collected is selected on the setting form 604. HTML and TEXT files are selected by default. Other than this, for example, Portable
Document format file, document file created by word processor software,
Attachments of various general-purpose file formats such as files created by spreadsheet software and presentation data files created by presentation data creation software can be selected as collection targets.

(5) Acquisition request: The timeout time and the number of retries for URL acquisition are specified on the "acquisition request" setting form 605. (6) Authentication Information: On the “authentication information” setting form 606, the authentication information required when accessing a URL that requires user authentication is set.

(7) Collection site: The following items are set on the "collection site" setting form 607.

Setting of starting URL: The URL which is the starting point of collection is specified. In this embodiment, a maximum of 100 excluding half-width characters and special characters is set on the setting field of the origin URL.
You can enter up to 0 characters.

Setting the upper limit of the number of link stages: Designate the maximum number of link stages from the collection origin URL. -Upper limit setting of the number of files: Specify the maximum number of files to be collected. -Re-collection setting: Origin U for the same collection task to be set
Specify the behavior when recollecting from RL. There are three modes for recollection ("collect all files newly", "collect only uncollected files", "collect only updated files (with deletion function)", and select any one mode. Can be specified.

Collection range: When "collect only in domain" is specified, collection is performed only in the domain of the origin URL. The default is no specification. If you select "Collect only in domain" from the check box, the source URL set in the field for setting the source URL will be documented from the relevant information site on the Internet / Intranet 30 only within that domain. To collect. The collection process at this time will be described later.

URL pattern to be collected: Designate a character string pattern of URLs to be collected. Among the URLs to be collected, only URLs that include the specified character string pattern are collected.

URL pattern not to be collected: Designate the character string pattern of the URL that should be excluded from the collection target. Among the URLs to be collected, the URL including the specified character string pattern is not collected.

In each recollection mode, the "collect all files newly" mode is the U
It means a mode of recollecting all RL. That is, regardless of whether it has been acquired in the past, it is newly collected from the origin URL. The “collect only uncollected files” mode is a mode in which previously collected URLs are not collected.
The "collect only updated files (with deletion function)" mode is a mode in which only the document files that have been updated from the previous collection point are collected. The document whose update is to be checked is the URL collected in the previous collection more than the specified number of days before the current collection. The value entered in the "Update check frequency" field is the specified number of days. The default number of designated days is "1 day". If the URL collected in the previous collection more than the specified number of days before the present is deleted from the corresponding information site on the Internet / Intranet 30 and the document file cannot be acquired from the URL, The registration information corresponding to the document file is deleted from the knowledge DB 131.

FIG. 11 shows an example of the status screen. For example, selecting the “status” anchor 503 on the list screen or the setting screen causes a transition to the status screen of the currently displayed task. The "setting name", "knowledge database name", "state", etc. of the task are displayed on this status screen. Next to the “setting name”, a button 701 for instructing start / stop (restart / interrupt) of collection is displayed. In "Status", the status of the task (collecting / finished / not collected / suspended / abnormally terminated) is displayed, and further, the starting URL of the task, the number of link stages collected (current / maximum / upper limit), and the number of files The status information 702 such as (current / total / upper limit) is also displayed.

When the status of the task is not collected, finished, or suspended, the button 701 displays "collection started", as shown in FIG. "Start collection" button 70
When 1 is selected, the execution of the task is started by the web collection module 111. If the current state is ended or suspended, recollection will be started. If the current status of the task is being collected,
As shown in FIG. 12, the button 701 displays “collection interruption”. If you select the “Collect collection” button 701,
Collection tasks that are running are suspended. The list of URLs collected before the interruption is held in the results file 204.

Next, with reference to FIG. 13 to FIG. 15, the interruption and resumption (recollection) operation of the collection process by the button operation on the status screen will be specifically described.

The suspend / resume function is a function used when the Web collection processing in progress is temporarily stopped and restarted at a later date. W by Web collection module 111
The eb collection process is performed by the internet / intranet 30
Each time a document file is acquired from the URL, the URL, a status code indicating success or failure, and the like are written in the result file 204 as the progress status information. If the acquired document file has a URL of the link destination, the URL is added to the result file 204, and the process is repeated until there is no uncollected URL in the result file 204. If the "collection suspension" button 701 is pressed during this collection processing, the Web collection task in execution is suspended at that point. Then, later, when the "collection start" button 701 is pressed with the interrupted Web collection task selected, the interrupted Web collection task is restarted and recollection is started. For example, when the “collect only uncollected files” mode is selected as the recollection mode, the uncollected URs held in the result file 204 are collected.
The collection process will be restarted from L.

FIG. 14 shows an example of a URL list held as progress status information in the result file 204. As shown in the figure, in the result file 204, in addition to the “URL” of each URL to be acquired, “collection date”, “collection date and time”, “status code”, “starting URL”, and “number of link stages” are included. Will be recorded. "Number of links" is
Indicates the number of links in the acquisition target URL when viewed from the starting URL. For the uncollected URLs, the information of “collection date”, “collection date and time”, and “status code” is blank.
This makes it possible to detect the restart point when the Web collection process is interrupted midway. The "status code" indicates the success or failure of the collection process, which is an HTTP status code returned from the partner web server from which the web page is acquired. The type of this HTTP status code is "200" indicating that the file acquisition was successful (normal end), "403" indicating an error due to the access authority, and the corresponding file cannot be found (for the request to acquire the deleted document file). “Response”) includes an error “404”, a timeout error “408”, and the like.

The flowchart of FIG. 15 shows a series of processing procedures from the start of Web collection to the interruption thereof.

The Web collection module 111 first
The document file specified by the origin URL is acquired (step S121), and the result (collection date, collection date and time, status code, origin URL, link stage number, URL (same as origin URL) is registered in the result file 204. The progress status information is updated (step S122) If the acquisition is successful, the knowledge DB 13 is acquired from the collected document files.
The attribute information to be registered in 1 is acquired (step S12
2) Next, text extraction processing (text extraction and detection of link destination URL in the case of HTML file) is executed (step S124). When the link destination URL is detected, the URL, the starting point URL, and the link stage number described above are additionally registered in the URL list of the result file 204 (step S125). After that, a process of describing the attribute information and the text data on the registration file is performed (step S126). Here, it is determined whether or not there is an interruption instruction (step S127). If there is no interruption instruction (NO in step S127), it is determined whether or not the uncollected URL is registered in the result file 204.
It is determined whether the collection process is completed (step S
128).

If uncollected URLs are registered in the result file 204 (NO in step S128), the processes from step S121 described above are repeated. That is, the document file specified by the URL added to the result file 204 is acquired (step S121),
The result (collection date, collection date and time, status code) is registered in the result file 204 to update the progress status information (step S122). If the acquisition is successful, the attribute information to be registered in the knowledge DB 131 is acquired from the collected document file (step S122), and then the text extraction process (text extraction and detection of link destination URL in the case of HTML file) is executed. (Step S12
4). If the linked URL is detected, its UR
L, the starting URL and the number of links are the result file 20
4 is additionally registered in the URL list (step S12).
5). After that, a process of writing the attribute information and the text data on the registration file 114 is performed (step S12).
6). Such Web collection processing is performed by the setting file 20.
It is recursively executed until the collection is completed within the collection range of the Web collection condition set to 3. When the "collection suspension" button 701 is pressed during this Web collection, the management interface 112 is accessed through the control file 202.
Issues an interruption instruction to the Web collection module 111 (YES in step S127), and the Web collection task being executed by the Web collection module 111 is interrupted.

Here, the Web collection module 11 when "collection only within domain" is set as the collection range
The Web collection process executed in 1 will be described with reference to FIG.

When "Setting anchor" 502 is clicked on the list screen shown in FIG. 9, the list screen shown in FIG. 9 is changed to the setting screen shown in FIG. 10 as shown in the screen transition diagram of FIG. , On the setting screen, "collection site"
By setting the setting form 607, the setting file 203 including the setting contents of the “collection site” setting form 607 is created (steps S201, 20).
2). In this embodiment, in the “collection site” setting form 607, in the setting field of the origin URL,
Set the starting URL of the site to be collected, and specify "Collect only within domain" in the checkbox of the collection range. At this time, in addition to the setting of "collect only in the domain", the pattern setting of the collected URL, the URL not collected
Multiple types of collection ranges can be set at the same time, such as the pattern setting of, the upper limit setting of the number of document files to be collected, the upper limit setting of the number of link stages, the setting of the file format to be collected, etc. No particular reference is made to the setting of the collection range other than the setting of "collect only in domain".

After setting the "collection site" setting form 607 on the setting screen shown in FIG. 10 and then clicking the "status anchor" 503 on the setting screen, the screen transition shown in FIG. As shown in FIG. 15, when the setting screen shown in FIG. 10 transits to the situation screen shown in FIG. 11 (step S203) and the “collection start” button 701 is selected on the situation screen, the situation shown in FIG. Web
The collection process is started, and in the Web collection process,
Web collection can be performed by narrowing the collection range according to the setting contents of the “collection site” setting form 607.

In order to narrow down the collection range, in this embodiment, collection processing is performed only within the domain of the origin URL. In the collection processing only within the domain, the starting point URL is read by referring to the setting file 203 (steps S204 to S206), and it is determined whether the collection only within the domain is set (step S207).

Here, if the collection is not set only within the domain (NO in step S207), the collection process from the origin URL is started (step S209), and the origin U
The collection shown in FIG. 15 is recursively executed for all the link destinations starting from the RL (step S210) (NO in steps S206 and S207, S20).
9, S210 NO, S206, ..., S210 YE
S).

If collection is set only in the domain (YES in step S207), all the sites in which the link to be traced is in the domain are set as collection sites in FIG. Collection shown in is collected recursively (steps S206 and S2).
YES at 07, YES at S208, S209, S210
No, S206, ..., S208 NO, S210 Y
ES).

In this way, the Web collection module 1
When the setting file 203 describes the setting information “collect only in the domain” in the setting file 203, only the domain of the set origin URL is targeted according to the setting content of the collection site setting screen and the network Collect the document information from. This allows knowledge information to be efficiently collected by narrowing down the sites to be collected within the domain, and shared knowledge information can be easily and quickly collected efficiently within a limited network range. .
Further, in addition to setting the collection range within the domain, the pattern setting of the URL to be collected, the pattern setting of the URL not to be collected, the upper limit value setting of the number of document files to be collected, the upper limit value setting of the link stage number, and the collecting When any or all of multiple types of collection ranges are set at the same time, such as file format settings, it is possible to collect by combining only the collection within the domain and other collection conditions (such as URL pattern matching). The desired knowledge information can be more appropriately narrowed down and collected, whereby a higher-level knowledge database can be constructed easily, quickly and efficiently.

Since all the functions of the knowledge information collecting system of this embodiment are realized by a computer program, the computer program is stored in a computer-readable storage medium, and the computer program is executed through the storage medium. It is possible to easily obtain the same effect as that of the present embodiment only by installing the program in a normal computer that can be connected to a computer network and executing the program.

Further, the present invention is not limited to the above-mentioned embodiment, but can be variously modified in an implementation stage without departing from the gist thereof. Furthermore, the embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the problem described in the section of the problem to be solved by the invention can be solved, and the effect described in the section of the effect of the invention can be solved. When the above is obtained, the configuration in which this constituent element is deleted can be extracted as the invention.

[0087]

As described above, according to the present invention,
Various types of document files having different file formats can be efficiently collected as knowledge information from the Internet.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a knowledge information collecting system according to an embodiment of the present invention.

FIG. 2 is an exemplary block diagram showing the functional configuration of a Web collection module provided in the knowledge information collection system of the same embodiment.

FIG. 3 is a diagram showing an example of an output format of a registration file used in the knowledge information collecting system of the embodiment.

FIG. 4 is an exemplary view showing another example of an output format of a registration file used in the knowledge information collecting system of the embodiment.

FIG. 5 is an exemplary flowchart showing a processing procedure of a Web collection module provided in the knowledge information collection system of the embodiment.

FIG. 6 is an exemplary flowchart showing a processing procedure of a registration module provided in the knowledge information collecting system of the embodiment.

FIG. 7: We in the knowledge information collecting system of the same embodiment
The figure which shows the relationship between b content and its registration processing.

FIG. 8 is a diagram for explaining three screens provided by the knowledge information collecting system of the embodiment and their transitions.

FIG. 9 is an exemplary view showing an example of a list screen used in the knowledge information collecting system of the embodiment.

FIG. 10 is an exemplary view showing an example of a setting screen used in the knowledge information collecting system of the embodiment.

FIG. 11 is an exemplary view showing an example of a situation screen used in the knowledge information collecting system of the embodiment.

FIG. 12 is an exemplary view showing an example of a situation screen used in the knowledge information collecting system of the embodiment.

FIG. 13 is an exemplary view for explaining a suspending / resuming operation of the Web collecting process in the knowledge information collecting system of the embodiment.

FIG. 14 is a diagram showing an example of contents of a result file provided in the knowledge information collecting system of the embodiment.

FIG. 15 is an exemplary flowchart showing a processing procedure from the start to the interruption of the Web collection processing in the knowledge information collection system of the embodiment.

FIG. 16 is an exemplary flowchart showing the procedure of the “collection only in domain” setting and the Web collection processing associated with the setting in the knowledge information collecting system of the embodiment.

[Explanation of symbols]

11 ... Web collection system 12 ... Registration module 13 ... Knowledge engine 30 ... Internet / Intranet 111 ... Web collection module 112 ... Management interface 113 ... Registration directory 131 ... Knowledge database 132 ... Search engine 201 ... Lock file 202 ... control file 203 ... Setting file 204 ... Result file 205 ... log file 301 ... Collection control unit 302 ... Attribute extraction unit 303 ... Text extraction unit 304 ... Format conversion unit 607 ... "Collection site" setting form

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Koji Shioda             2-9 Suehiro-cho, Ome City, Tokyo Stock Market             Company Toshiba Ome Factory F-term (reference) 5B075 ND03 NK37 NK44 QP01

Claims (6)

[Claims] 1. A knowledge information collecting system for collecting document information to be registered in a knowledge database from a network based on the information set by the setting means, wherein the setting means has a domain of a starting URL. An information collecting means for collecting the document information to be registered in the knowledge database, which includes means for setting collection, and when the in-domain collection is set by the setting means, only in the set domain. A knowledge information collecting system characterized by being equipped. 2. A knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, having a starting point URL setting item and a collection range narrowing setting item for defining a collection range within a domain. Graphic user interface, and the starting URL in the graphic user interface
And when the narrowing of the collection range is set together,
The document information is collected from the network only within the domain of the set origin URL, and the origin UR is collected.
When L is set and the narrowing down of the collection range is not set, a collecting means for collecting the document information from the network for a site including a link destination outside the domain of the set origin URL is provided. A knowledge information collection system characterized by what is done. 3. In a knowledge information collecting system for collecting document information to be registered in a knowledge database from a network, a starting URL setting field for setting a site for collecting document information to be registered in the knowledge database. , And a setting screen display means for displaying a collection site setting screen including items defining the collection range, and a setting file for creating a setting file based on the setting contents on the setting screen displayed by the setting screen display means. And a collection means for collecting document information to be registered in the knowledge database from the network based on the contents of the setting file created by the setting file creating means. Is the setting field of the origin URL and multiple types of setting items including the setting items whose collection range is within the domain. Has, the information collecting unit, the configuration file, the starting point UR
When L and the content of which the collection range is within the domain are set, the document information is downloaded from the network according to the setting content of the collection site setting screen, only within the domain of the set origin URL. Knowledge information collection system characterized by collecting. 4. The collection site setting screen displayed by the setting screen display means includes items for setting a pattern of URLs to be collected and URLs not to be collected, as setting items other than the setting items having the collection range within the domain. The knowledge information collecting system according to claim 3, including at least one of an item for setting a pattern, an item for setting a file format to be collected, an item for setting an upper limit of links to be collected, and an item for setting an upper limit of the number of files to be collected. . 5. In a knowledge information collecting method for collecting document information to be registered in a knowledge database from a network, a setting field of a starting point URL and a setting of a collecting site having an item for specifying a collecting range only within a domain. The setting screen displaying step of displaying a screen, the setting file creating step of creating a setting file based on the contents set in the setting screen displaying step, and the contents of the setting file created in the setting file creating step are also included. And an information collecting step of collecting document information to be registered in the knowledge database from a network, wherein the information collecting step includes a starting point U in the setting file.
Knowledge information collection characterized by collecting the document information from the network only within the domain of the set origin URL when the RL and the information whose collection range is within the domain are set Method. 6. A knowledge information collecting method for collecting document information to be registered in a knowledge database from a network, wherein a starting URL setting field for setting a site for collecting document information to be registered in the knowledge database. , And a setting screen displaying step for displaying a collection site setting screen including items for defining the collection range, and a setting file for creating a setting file based on the setting contents on the setting screen displayed by the setting screen displaying step. There is a creating step and a collecting step of collecting document information to be registered in the knowledge database from a network based on the contents of the setting file created in the setting file creating step. , The setting field of the origin URL, and the setting items that the collection range is within the domain Along with, set items for setting the pattern of URLs to be collected, items for setting patterns of URLs not to be collected, items for setting the file format to be collected, items for setting the upper limit of links to be collected, and upper limits for the number of files to be collected At least one of the items is included, and in the collecting step, the origin UR is included in the configuration file.
When L and the content of which the collection range is within the domain are set, the document information is downloaded from the network according to the setting content of the collection site setting screen, only within the domain of the set origin URL. A knowledge information collecting method characterized by collecting.