CN114329137A - Crawler method and device, electronic equipment and storage medium - Google Patents

Abstract

The invention provides a crawler method, a crawler device, electronic equipment and a storage medium, and the technical scheme is as follows: compiling a crawler task code corresponding to a website to be crawled and then publishing the crawler task code to a project path of a crawler system; the website information to be captured is sent to a message queue of a crawler system; the website information to be grabbed comprises an initial url and a full-limited name of a crawler task class defined in the crawler task code; acquiring website information to be captured from a message queue of a crawler system, capturing a website page corresponding to an initial url in the website information to be captured according to a full limit name of a crawler task class defined in a crawler task code, performing page analysis on the website page corresponding to the initial url, and storing a page analysis result of the website page corresponding to the initial url. The invention can realize full configuration and customization operation of the crawler code without shutdown, and does not need shutdown for redeployment.

Description

A crawler method, device, electronic device and storage medium

technical field

The invention relates to the technical field of network search, in particular to a crawler method, device, electronic device and storage medium.

Background technique

The classic crawler system is generally divided into four modules: task scheduling module, page crawling module, page parsing module and data storage module. In the classic crawler system, the task scheduling module generally uses the message queue to realize the storage and scheduling of the crawler url link. The initial url is generally read into the message queue through program hard coding, database, and configuration file, and then enters the page crawling module to crawl The parsed page then enters the page parsing module, and the final parsed data enters the data storage module for storage. The above business logic for crawling web page data and related various attribute configurations are written when the crawler system is developed.

In traditional crawler systems, once the business logic code for crawling web page data is written, it can only be crawled according to the established logic. If the page structure of the crawled website changes or the user crawling logic and attributes (such as crawling fields, If the number of pages taken, crawling tools, data storage system, etc.) changes, the user can only re-develop the code, and then package and re-release the crawler system. In addition, if you want to crawl a new website, you also need to develop new code on the original project, shut down the production system, and then redeploy the system. For a large crawler system with dozens or even hundreds of crawler tasks, frequent downtime deployments are unacceptable.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a crawler method, device, electronic device and storage medium, which can realize full configuration and customized operation of crawler code without downtime, and do not need downtime for redeployment.

In order to achieve the above object, the present invention provides the following technical solutions:

A crawler method, applied to a crawler system, includes:

Compile the crawler task code corresponding to the website to be crawled and publish it to the project path of the crawler system;

Send the website information to be crawled to the message queue of the crawler system; the website information to be crawled includes the initial url and the fully qualified name of the crawler task class defined in the crawler task code;

The information of the website to be crawled is obtained from the message queue of the crawler system, and according to the fully qualified name of the crawler task class defined in the crawler task code, the website page corresponding to the initial url in the information of the website to be crawled is crawled. The website page corresponding to the initial url is subjected to page parsing, and the page parsing result of the website page corresponding to the initial url is stored.

A crawler device, comprising: a task scheduling module, a page grabbing module, a page parsing module, a data storage module, a message starting module, and a code publishing module;

The code publishing module is used to compile and publish the crawler task code corresponding to the website to be crawled to the project path of the crawler system;

The message startup module is used to send the information of the website to be crawled to the message queue maintained by the task scheduling module; the information of the website to be crawled includes the initial url and the crawler task class defined in the crawler task code. fully qualified name;

The page crawling module is used to obtain the information of the website to be crawled from the message queue maintained by the task scheduling module, and according to the fully qualified name of the crawler task class defined in the crawler task code, crawling the website to be crawled The website page corresponding to the initial url in the information;

The page parsing module is used to obtain the information of the website to be crawled from the message queue maintained by the task scheduling module, and according to the fully qualified name of the crawler task class defined in the crawler task code, to the corresponding initial url. Web page analysis;

The data storage module is used to obtain the information of the website to be crawled from the message queue maintained by the task scheduling module, and according to the fully qualified name of the crawler task class defined in the crawler task code, store the page parsing module pair. The page parsing result of the website page corresponding to the initial url.

An electronic device comprising: at least one processor, and a memory connected to the at least one processor through a bus; the memory stores one or more computer programs executable by the at least one processor; the The steps in the above crawler method are implemented when at least one processor executes the one or more computer programs.

A computer-readable storage medium storing one or more computer programs, when the one or more computer programs are executed by a processor, implement the steps in the above-mentioned crawling method.

As can be seen from the above technical solutions, in the present invention, the crawler task code corresponding to the website to be crawled is compiled and published under the project path of the crawler system; The website page corresponding to the initial url in the website information is crawled, and the website page corresponding to the initial url is parsed, and the page analysis result of the website page corresponding to the initial url is stored. It can be seen that in the present invention, through the release of the crawler task code corresponding to the website to be crawled and the fully qualified name of the crawler task class defined by the crawler task code, the full configuration and customization of the crawler code can be realized without downtime. Operation without downtime for redeployment.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

1 is a flowchart of a crawler method according to an embodiment of the present invention;

Fig. 2 is the flow chart of the crawler method according to the second embodiment of the present invention;

Fig. 3 is the flow chart of the third embodiment of the present invention crawler method;

Fig. 4 is the flow chart of the four crawler method according to the embodiment of the present invention;

5 is a schematic structural diagram of a crawler device provided by an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The terms "first", "second", "third", "fourth", etc. (if present) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can, for example, be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units expressly listed, but may include steps or units not expressly listed or for such process, method, product or Other steps or units inherent to the device.

The technical solutions of the present invention will be described in detail below with specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

Referring to FIG. 1, FIG. 1 is a flowchart of a crawler method according to an embodiment of the present invention. The method is applied to a crawler system, as shown in FIG. 1, and mainly includes the following steps:

Step 101: Compile the crawler task code corresponding to the website to be crawled and publish it to the project path of the crawler system;

Step 102, sending the information of the website to be crawled to the message queue of the crawler system; the information of the website to be crawled includes the initial url and the fully qualified name of the crawler task class defined in the crawler task code;

Step 103, obtaining the information of the website to be crawled from the message queue of the crawler system, and according to the fully qualified name of the crawler task class defined in the crawler task code, crawling the website page corresponding to the initial url in the information of the website to be crawled, Perform page parsing on the website page corresponding to the initial url, and store the page parsing result of the website page corresponding to the initial url.

It can be seen from the method shown in FIG. 1 that in this embodiment, the crawler task code corresponding to the website to be crawled is compiled and released to the project path of the crawler system; according to the crawler task class defined in the crawler task code The fully qualified name of the website page corresponding to the initial url in the website information to be crawled is crawled, the website page corresponding to the initial url is parsed, and the page parsing result of the website page corresponding to the initial url is stored. In this embodiment, through the release of the crawler task code corresponding to the website to be crawled and the fully qualified name of the crawler task class defined by the crawler task code, the fully configured and customized operation of the crawler code can be realized without downtime. Requires downtime to redeploy.

Referring to FIG. 2, FIG. 2 is a flowchart of the second embodiment of the present invention, a crawler method. The method is applied to a crawler system, as shown in FIG. 2, and mainly includes the following steps:

Step 201: Compile the crawler task code corresponding to the website to be crawled and publish it to the project path of the crawler system;

In actual implementation, the crawler task code corresponding to the website to be crawled can be compiled into a class file, and the class file can be published to the project path of the crawler system.

Step 202, sending the information of the website to be crawled to the message queue of the crawler system; the information of the website to be crawled includes the initial url and the fully qualified name of the crawler task class defined in the crawler task code; the crawler task code The fully qualified name of the crawler task class defined in includes the fully qualified name of the first crawler task class.

In this embodiment, the information of the website to be crawled is sent as the message body to the message queue maintained by the crawler system. By including various information related to page crawling in the information of the website to be crawled, it is used for subsequent page crawling operations. , page parsing operations, and data storage operations.

Step 2031: Obtain the information of the website to be crawled from the message queue of the crawler system;

Step 2032: According to the fully qualified name of the first crawler task class defined in the crawler task code, generate the first crawler task class instance object through the java reflection mechanism, and call the page crawling method of the first crawler class task instance object to be crawled. Take the website page corresponding to the initial url in the website information for crawling;

In practical applications, the core of the java reflection mechanism is to dynamically load the class and obtain the detailed information of the class when the program is running, so as to operate the attributes and methods of the class or object. The essence is that after the JVM obtains the class object, it decompiles the class object. , so as to obtain various information about the object.

In this embodiment, the class file obtained after compiling the crawler task code corresponding to the website to be crawled is published to the project path of the crawler system, so that the class object in the class file (for example, the above-mentioned class object can be obtained through the java reflection mechanism later) The first crawler task class instance object), decompile the class object, so that various information of the class object can be obtained, including the page grabbing method of the first crawler task class instance object, and then the page grabbing method can be used to The website page corresponding to the initial url is crawled.

In an embodiment of the present invention, the information on the website to be crawled may further include a crawling tool. Developers can select a crawling tool suitable for the website to be crawled according to actual needs, such as one of the httpClient client, Chrome headless browser, phantomJs and other crawling tools.

The page crawling method that calls the first crawler class task instance object to crawl the website page corresponding to the initial url in the website information to be crawled may specifically include:

Take the crawler in the information of the website to be crawled as the input parameter of the page crawling method of the first crawler task instance object for specifying the crawler, and call the page crawling method of the first crawler task instance object to be crawled Take the website page corresponding to the initial url in the website information for crawling.

Here, the crawling tool in the information of the website to be crawled is used as the input parameter for specifying the crawling tool in the page crawling method of the first crawler task instance object, so that when the page crawling method is called, the page crawling method can be used. The crawling tool performs page crawling on the website page corresponding to the initial url.

In another embodiment of the present invention, the information on the website to be crawled further includes the number n of crawled pages. In practical applications, some web pages have multiple pages. For example, there will be hundreds of pages for a relatively popular post. In this case, you can choose not to crawl all pages, but only the first few pages. Therefore, , the number of pages n to be crawled can be specified in the information of the website to be crawled to reduce resource occupation caused by page crawling, where the value of n can be determined according to requirements, for example, it can be set to 10.

The page crawling method that calls the first crawler class task instance object to crawl the website page corresponding to the initial url in the website information to be crawled may specifically include:

Taking the number of pages to grab as the input parameter of the page grabbing method of the first crawler task instance object for specifying the maximum number of grabbed pages, and calling the page grabbing method of the first crawler task instance object to grab the website The website page corresponding to the initial url in the information is crawled.

Here, the number of crawled pages is used as an input parameter of the page crawling method of the first crawler task instance object for specifying the maximum number of crawled pages, so that when the page crawling method is called, if the initial url If the corresponding website page has multiple pages, only the first n pages will be crawled.

The above step 2032 is a specific refinement of “crawl the website page corresponding to the initial url in the website information to be crawled according to the fully qualified name of the crawler task class defined in the crawler task code” in step 103 shown in FIG. 1 .

Step 2033: Perform page parsing on the website page corresponding to the initial url according to the fully qualified name of the crawler task class defined in the crawler task code, and store the page parsing result of the website page corresponding to the initial url.

The above steps 2031 to 2033 are specific refinements of step 103 shown in FIG. 1 .

According to the method shown in FIG. 2, it can be seen that in this embodiment, the crawler task code corresponding to the website to be crawled is compiled and released to the project path of the crawler system; the information of the website to be crawled is sent to the message queue of the crawler system , generate the first crawler task class instance object through the java reflection mechanism, call the page crawling method of the first crawler class task instance object to crawl the website page corresponding to the initial url in the website information to be crawled, and fetch the website page corresponding to the initial url in the website information. Perform page parsing on the website page corresponding to the url, and store the page parsing result of the website page corresponding to the initial url. In this embodiment, through the release of the crawler task code corresponding to the website to be crawled and the fully qualified name of the crawler task class defined by the crawler task code, the fully configured and customized operation of the crawler code can be realized without downtime. Requires downtime to redeploy. In addition, in this embodiment, by including a crawling tool in the message of the website to be crawled, the page crawling module is instructed to use the crawling tool to crawl the page of the website to be crawled, so as to avoid the inappropriate crawling tool. In addition, the information of the website to be crawled includes the number of pages to be crawled, so as to control the number of pages crawled by the page crawling module for the website page corresponding to the initial url, and reduce the number of pages crawled due to too many pages. resource usage.

Referring to FIG. 3, FIG. 3 is a flowchart of the third embodiment of the present invention, a crawler method. The method is applied to a crawler system, as shown in FIG. 3, and mainly includes the following steps:

Step 301: Compile and publish the crawler task code corresponding to the website to be crawled under the project path of the crawler system;

In actual implementation, the crawler task code corresponding to the website to be crawled can be compiled into a class file, and the class file can be published to the project path of the crawler system.

Step 302, sending the information of the website to be crawled to the message queue of the crawler system; the information of the website to be crawled includes the initial url and the fully qualified name of the crawler task class defined in the crawler task code; the crawler task code The fully qualified name of the crawler task class defined in includes the fully qualified name of the second crawler task class;

In this embodiment, by including various information related to page crawling in the information of the website to be crawled, a basis is provided for subsequent page crawling operations, page parsing operations, and data storage operations.

Step 3031: Obtain the information of the website to be crawled from the message queue of the crawler system;

Step 3032, according to the fully qualified name of the crawler task class defined in the crawler task code, grab the website page corresponding to the initial url in the website information to be crawled;

Step 3033: According to the fully qualified name of the second crawler task class defined in the crawler task code, generate the second crawler task class instance object through the java reflection mechanism, and call the page parsing method of the second crawler class task instance object to analyze the Perform page analysis on the website page corresponding to the initial url;

In practical applications, the core of the java reflection mechanism is to dynamically load the class and obtain the detailed information of the class when the program is running, so as to operate the attributes and methods of the class or object. The essence is that after the JVM obtains the class object, it decompiles the class object. , so as to obtain various information about the object.

In this embodiment, the class file obtained after compiling the crawler task code corresponding to the website to be crawled is published to the project path of the crawler system, so that the class object in the class file (for example, the above-mentioned class object can be obtained through the java reflection mechanism later) The second crawler task class instance object), decompile the class object, so that various information of the class object can be obtained, including the page parsing method of the second crawler task class instance object, and then the page parsing method can be used to analyze the initial The website page corresponding to the url is parsed. Since the page parsing method is a parsing code generated specifically for the page structure of the website to be crawled, accurate parsing of the page can be achieved.

In an embodiment of the present invention, the information on the website to be crawled further includes a crawling field. In practical applications, there are two purposes of page crawling. One is to obtain the content of the page, and the other is to obtain the new url associated with the page for further crawling. For this, it is not necessary to parse all the fields in the page. , and only need to parse some fields in the page in a targeted manner to achieve the above purpose. Therefore, in this embodiment, it is possible to specify which fields need to be parsed in the information of the website to be crawled. The following fields are parsed: title, clicknum, reply, pubTime, author, content, img.

The page parsing method for invoking the second crawler task instance object to perform page parsing on the website page corresponding to the initial url may specifically include:

Use the grab field as an input parameter of the page parsing method of the second crawler task instance object for specifying the target parsing field, call the page parsing method of the second crawler task instance object, and parse the website corresponding to the initial url Field content for each fetched field in the page.

The grab field is used as the input parameter of the page analysis method of the second crawler task instance object for specifying the target analysis field, so that when the page grab method is called, only the grab field can be parsed .

Step 3034: Store the page parsing result of the website page corresponding to the initial url according to the fully qualified name of the crawler task class defined in the crawler task code.

It can be seen from the method shown in FIG. 3 that in this embodiment, the crawler task code corresponding to the website to be crawled is compiled and published under the project path of the crawler system; according to the crawler task class defined in the crawler task code The fully qualified name of the website page corresponding to the initial url in the website information to be crawled is to be crawled; according to the fully qualified name of the second crawler task class defined in the crawler task code, the second crawler task class instance is generated through the java reflection mechanism object, call the page parsing method of the second crawler class task instance object to perform page parsing on the website page corresponding to the initial url; store the page parsing result of the website page corresponding to the initial url. In this embodiment, through the release of the crawler task code corresponding to the website to be crawled and the fully qualified name of the crawler task class defined by the crawler task code, the fully configured and customized operation of the crawler code can be realized without downtime. Requires downtime to redeploy. In addition, in this embodiment, only some fields in the page are parsed, thereby reducing resource occupation caused by page parsing.

Referring to FIG. 4, FIG. 4 is a flowchart of the fourth embodiment of the present invention, a crawler method. The method is applied to a crawler system, as shown in FIG. 4, and mainly includes the following steps:

Step 401: Compile the crawler task code corresponding to the website to be crawled and publish it under the project path of the crawler system;

In actual implementation, the crawler task code corresponding to the website to be crawled can be compiled into a class file, and the class file can be published to the project path of the crawler system.

Step 402: Send the information of the website to be crawled to the message queue of the crawler system; the information of the website to be crawled includes the initial url and the fully qualified name of the crawler task class defined in the crawler task code; the crawler task code The fully qualified name of the crawler task class defined in includes the fully qualified name of the third crawler task class;

In this embodiment, by including various information related to page crawling in the information of the website to be crawled, a basis is provided for subsequent page crawling operations, page parsing operations, and data storage operations.

Step 4031: Obtain the information of the website to be crawled from the message queue of the crawler system;

Step 4032: According to the fully qualified name of the crawler task class defined in the crawler task code, grab the website page corresponding to the initial url in the website information to be crawled, and perform page analysis on the website page corresponding to the initial url;

Step 4033: According to the fully qualified name of the third crawler task class defined in the crawler task code, generate the third crawler task class instance object through the java reflection mechanism, and call the page storage method of the third crawler class task instance object to describe the The page parsing result of the website page corresponding to the initial url is stored.

In practical applications, the core of the java reflection mechanism is to dynamically load the class and obtain the detailed information of the class when the program is running, so as to operate the attributes and methods of the class or object. The essence is that after the JVM obtains the class object, it decompiles the class object. , so as to obtain various information about the object.

In this embodiment, the class file obtained after compiling the crawler task code corresponding to the website to be crawled is published to the project path of the crawler system, so that the class object in the class file (for example, the above-mentioned class object can be obtained through the java reflection mechanism later) The third crawler task class instance object), decompile the class object, so as to obtain various information of the class object, including the page storage method of the third crawler task class instance object, and then use the page storage method to The page parsing result of the website page corresponding to the url is stored.

In an embodiment of the present invention, the website information to be crawled further includes data identification generation rules and target storage system information, wherein the data identification generation rules include fields involved in generating data identifications and a method for generating data identifications. The method for generating the data identification can be determined according to specific needs. For example, the fields involved in generating the data identification can be specified as url, title, and pubTime, and the method for generating the data identification can be: The preset sequence is spliced together to generate the MD5 value corresponding to the spliced string, and the MD5 value is used as the generated data identifier. The preset sequence may be determined according to actual requirements, for example, the preset sequence may be: content of the url field, content of the title field, and content of the pubTime field.

In practical applications, data storage needs to be performed after the page is captured. When data storage is performed, a data identifier can be generated for the data to be stored, and then the data identifier and the data to be stored are stored correspondingly, so as to facilitate the subsequent possible occurrence of For operations such as data query, for this purpose, data identification generation rules may be specified in the information of the website to be crawled, and in addition, the target storage system information of the data storage may be specified in the information of the website to be crawled. Here, the target storage system information may specifically include the following information: the type of the target storage system (for example, mysql), the link address (identified by url), and the user name (username) and password (password) for logging in to the target storage system. .

In this embodiment, calling the page storage method of the third crawler task instance object to store the page parsing result of the website page corresponding to the initial url may specifically include:

Determine the field content of each field involved in generating the data identifier in the page parsing result;

Determine the data identifier corresponding to the page parsing result according to the field content of each field involved in generating the data identifier and the method for generating the data identifier;

The page storage method of the third crawler class task instance object is called, and the page parsing result and its corresponding data identifier are stored in the target storage system.

According to the method shown in FIG. 4, it can be seen that in this embodiment, the crawler task code corresponding to the website to be crawled is compiled and published under the project path of the crawler system; according to the crawler task class defined in the crawler task code The fully qualified name of the website page corresponding to the initial url in the website information to be crawled is crawled and the website page corresponding to the initial url is parsed; according to the third crawler defined by the crawler task code in the website information to be crawled The fully qualified name of the task class, the third crawler task class instance object is generated through the java reflection mechanism, and the page storage method of the third crawler class task instance object is called to store the page parsing result of the website page corresponding to the initial url, thereby completing The page crawling process for the site to be crawled. In this embodiment, through the release of the crawler task code corresponding to the website to be crawled and the fully qualified name of the crawler task class defined by the crawler task code, the fully configured and customized operation of the crawler code can be realized without downtime. Requires downtime to redeploy. In addition, in this embodiment, only some fields in the page are parsed, thereby reducing resource occupation caused by page parsing. In addition, by generating a data identifier corresponding to the page parsing result, the data identifier and the page parsing result are stored in the target storage system, so that subsequent data operations such as data query can be facilitated.

The crawler method provided by the embodiments of the present invention has been described in detail above. It should be noted that, in the above four embodiments, the fully qualified name of the first crawler task class, the fully qualified name of the second crawler task class, and the third crawler task class The fully qualified names of the classes can be the same or different. Among them, when the three fully qualified names are the same, it means the same crawler task class. In this case, the crawler task class defines the page grabbing method, page parsing method, and page storage method at the same time. When an instance object is created, only one instance object of the crawler task class can be generated, and page crawling, page parsing, and data storage can be performed by calling the page crawling method, page parsing method, and page storage method in the instance object. operate. When the three fully qualified names are different from each other, it means that they are different crawler task classes, that is, three crawler task classes are defined in the crawler task code, corresponding to the first crawler task class (which defines the page crawling method) and the second crawler task class. class (defines the page parsing method), the third crawler task class (defines the page storage method).

The following is a specific example of the information of the website to be crawled as the message body. The information of the website to be crawled includes the following information: initial url (http://bbs.tianya.cn/post-travel-862656-1.shtml), Fully qualified name of the shared crawler task class (com.jd.crawler.TianyaProcessor), crawler (Chrome), number of crawled pages (10), crawling fields (title, clicknum, reply, pubTime, author, content, img), data identification generation rules (participating fields: url, title, pubTime, the data identification generation method is as in the previous example, and will not be repeated), target storage system information.

For the above example, when there is no crawler task code suitable for the website to be crawled in the crawler system, the page crawling of the website to be crawled can be achieved through the following process:

First, developers can generate crawler task code corresponding to the website to be crawled by modifying the existing crawler code or re-writing the crawler task code, and then compile and publish the crawler task code to the project path of the crawler system; The website to be crawled is sent to the message queue of the crawler system;

Next, the information of the website to be crawled can be obtained from the message queue of the crawler system, and according to the fully qualified name of the crawler task class (com.jd.crawler.TianyaProcessor) in the information of the website to be crawled, an instance of the crawler task class can be generated through the java reflection mechanism The object TianyaProcessor takes the crawler Chrome and the number of pages to be crawled as input parameters, and calls the page parsing method of the crawler task class instance object TianyaProcessor to grab the initial url (http://bbs.tianya.cn/post-travel-862656- 1. shtml) the corresponding website page;

Next, take the grab fields (title, clicknum, reply, pubTime, author, content, img) as input parameters, call the page parsing method of the crawler task instance object TianyaProcessor to perform page parsing on the website page corresponding to the initial url to obtain Field content of the following fields: title, clicknum, reply, pubTime, author, content, img;

Finally, determine the field content of each field (url, title, pubTime) involved in generating the data identifier in the page parsing result of the website page corresponding to the initial url; The data identification method determines the data identification corresponding to the page parsing result; the page storage method of the TianyaProcessor of the crawler class task instance object is called to store the page parsing result and its corresponding data identification in the target storage system.

An embodiment of the present invention further provides a crawler method, which will be described in detail below with reference to FIG. 5 .

Referring to FIG. 5, FIG. 5 is a schematic structural diagram of a crawler device provided by an embodiment of the present invention. The device is applied to a crawler system, as shown in FIG. 5, including: a task scheduling module 501, a page grabbing module 502, a page parsing module 503, a data storage module 504, a message activation module 505, and a code release module 506;

The code publishing module 506 is used to compile and publish the crawler task code corresponding to the website to be crawled to the project path of the crawler system;

The message startup module 505 is used to send the information of the website to be crawled to the message queue maintained by the task scheduling module 501; the information of the website to be crawled includes the initial url and the crawler task defined in the crawler task code the fully qualified name of the class;

The page crawling module 502 is used to obtain the information of the website to be crawled from the message queue maintained by the task scheduling module 501, and according to the fully qualified name of the crawler task class defined in the crawler task code, crawling the to-be-crawled website Get the website page corresponding to the initial url in the website information;

The page parsing module 503 is used to obtain the information of the website to be crawled from the message queue maintained by the task scheduling module 501, and according to the fully qualified name of the crawler task class defined in the crawler task code, the initial url Perform page analysis on the corresponding website page;

The data storage module 504 is configured to obtain the information of the website to be crawled from the message queue maintained by the task scheduling module 501, and store the page parsing according to the fully qualified name of the crawler task class defined in the crawler task code. The module 503 analyzes the page of the website page corresponding to the initial url.

In the device shown in Figure 5,

The fully qualified name of the crawler task class defined in the crawler task code includes the fully qualified name of the first crawler task class;

The page crawling module 502, according to the fully qualified name of the crawler task class defined in the crawler task code, crawls the website page corresponding to the initial url in the information of the website to be crawled, including:

According to the fully qualified name of the first crawler task class defined in the crawler task code, the first crawler task class instance object is generated through the java reflection mechanism, and the page crawling method of the first crawler class task instance object is called to grab the website information Crawl the website page corresponding to the initial url in .

In the device shown in Figure 5,

The information on the website to be crawled further includes crawling tools;

The page grabbing module 502 calls the page grabbing method of the first crawler task instance object to grab the website page corresponding to the initial url in the website information to be grabbed, including:

The crawling tool is used as the input parameter for specifying the crawling tool in the page crawling method of the first crawler class task instance object, and the page crawling method of the first crawler class task instance object is called to be crawled in the website information. The website page corresponding to the initial url is crawled.

In the device shown in Figure 5,

The information of the website to be crawled further includes the number of crawled pages n;

The page grabbing module 502 calls the page grabbing method of the first crawler task instance object to grab the website page corresponding to the initial url in the website information to be grabbed, including:

Taking the number of pages to grab as the input parameter of the page grabbing method of the first crawler task instance object for specifying the maximum number of grabbed pages, and calling the page grabbing method of the first crawler task instance object to grab the website The website page corresponding to the initial url in the information is crawled.

In the device shown in Figure 5,

The fully qualified name of the crawler task class defined in the crawler task code includes the fully qualified name of the second crawler task class;

The page parsing module 503 performs page parsing on the website page corresponding to the initial url according to the fully qualified name of the crawler task class defined in the crawler task code, including:

According to the fully qualified name of the second crawler task class defined in the crawler task code, the second crawler task class instance object is generated through the java reflection mechanism, and the page parsing method of the second crawler class task instance object is called to correspond to the initial url page analysis of the website page.

In the device shown in Figure 5,

The information on the website to be crawled includes a crawling field;

The page parsing module 503 invokes the page parsing method of the second crawler task instance object to perform page parsing on the website page corresponding to the initial url, including:

Use the grab field as an input parameter of the page parsing method of the second crawler task instance object for specifying the target parsing field, call the page parsing method of the second crawler task instance object, and parse the website corresponding to the initial url Field content for each fetched field in the page.

In the device shown in Figure 5,

The fully qualified name of the crawler task class defined in the crawler task code includes the fully qualified name of the third crawler task class;

The data storage module 504, according to the fully qualified name of the crawler task class defined in the crawler task code, stores the page parsing result of the website page corresponding to the initial url by the page parsing module 503, including:

According to the fully qualified name of the third crawler task class defined in the crawler task code, the third crawler task class instance object is generated through the java reflection mechanism, and the page storage method of the third crawler class task instance object is called to correspond to the initial url The page parsing results of the website pages are stored.

In the device shown in Figure 5,

The website information to be crawled includes a data identification generation rule and a target storage system, wherein the data identification generation rule includes a field involved in generating a data identification and a method for generating a data identification;

The data storage module 504 calls the page storage method of the third crawler task instance object to store the page parsing result of the website page corresponding to the initial url, including:

Determine the field content of each field involved in generating the data identifier in the page parsing result;

Determine the data identifier corresponding to the page parsing result according to the field content of each field involved in generating the data identifier and the method for generating the data identifier;

The page storage method of the third crawler class task instance object is called, and the page parsing result and its corresponding data identifier are stored in the target storage system.

In the device shown in Figure 5,

The described fields that participate in generating the data identification include: url field, tilte field, pubTime field;

The method for generating a data identifier is as follows: splicing together the field contents of the fields involved in generating the data identifier according to a preset order, generating an MD5 value corresponding to the spliced character string, and using the MD5 value as the generated data identifier.

An embodiment of the present invention further provides an electronic device. As shown in FIG. 6 , the electronic device includes: at least one processor 601 and a memory 602 connected to the at least one processor 601 through a bus; the memory 602 stores There are one or more computer programs that can be executed by the at least one processor 601; the at least one processor 601 implements the one or more computer programs when executing the one or more computer programs. Steps in the crawler method.

An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores one or more computer programs, and the one or more computer programs are implemented by a processor when executed as shown in FIGS. 1-4 Steps in the crawler method shown in any of the flowcharts.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (12)

1. A crawler method is applied to a crawler system and is characterized by comprising the following steps:

compiling a crawler task code corresponding to a website to be crawled and then publishing the crawler task code to a project path of a crawler system;

the website information to be captured is sent to a message queue of a crawler system; the website information to be grabbed comprises an initial url and a full-limited name of a crawler task class defined in the crawler task code;

acquiring website information to be captured from a message queue of a crawler system, capturing a website page corresponding to an initial url in the website information to be captured according to a full limit name of a crawler task class defined in a crawler task code, performing page analysis on the website page corresponding to the initial url, and storing a page analysis result of the website page corresponding to the initial url.

2. The method of claim 1,

the fully qualified names of the crawler task classes defined in the crawler task codes comprise fully qualified names of a first crawler task class;

and capturing a website page corresponding to an initial url in the website information to be captured according to the full limit name of the crawler task class defined in the crawler task code, wherein the capturing comprises the following steps:

and generating a first crawler task class instance object through a java reflection mechanism according to the full limit name of the first crawler task class defined in the crawler task code, and calling a page grabbing method of the first crawler task class instance object to grab a website page corresponding to an initial url in the to-be-grabbed website information.

3. The method of claim 2,

the website information to be grabbed further comprises a grabbing tool;

the method for capturing the website page corresponding to the initial url in the website information to be captured by calling the page capturing method of the first crawler task instance object comprises the following steps:

and taking the grabbing tool as an input parameter of a specified grabbing tool of a page grabbing method of the first crawler task instance object, and calling the page grabbing method of the first crawler task instance object to grab the website page corresponding to the initial url in the website information to be grabbed.

4. The method of claim 2,

the website information to be grabbed also comprises a grabbing page number n;

the method for capturing the website page corresponding to the initial url in the website information to be captured by calling the page capturing method of the first crawler task instance object comprises the following steps:

and taking the snatching page number as an input parameter of the appointed maximum snatching page number of the page snatching method of the first crawler task instance object, and calling the page snatching method of the first crawler task instance object to snatch the website page corresponding to the initial url in the website information to be snatched.

5. The method of claim 1,

the fully qualified names of the crawler task classes defined in the crawler task codes comprise fully qualified names of a second crawler task class;

the page analysis module is used for carrying out page analysis on the website page corresponding to the initial url according to the fully-defined name of the crawler task class defined in the crawler task code, and comprises the following steps:

and generating a second crawler task class instance object through a java reflection mechanism according to the full limit name of the second crawler task class defined in the crawler task code, and calling a page resolution method of the second crawler task instance object to perform page resolution on the website page corresponding to the initial url.

6. The method of claim 5,

the website information to be grabbed comprises grabbing fields;

calling a page analysis method of a second crawler task instance object to perform page analysis on the website page corresponding to the initial url, wherein the page analysis method comprises the following steps:

and taking the captured fields as input parameters of a specified target analysis field of a page analysis method of a second crawler task instance object, calling the page analysis method of the second crawler task instance object, and analyzing the field content corresponding to each captured field in the website page corresponding to the initial url.

7. The method of claim 1,

the fully qualified names of the crawler task classes defined in the crawler task codes comprise fully qualified names of a third crawler task class;

storing the page analysis result of the website page corresponding to the initial url by the page analysis module according to the fully qualified name of the crawler task class defined in the crawler task code, wherein the page analysis result comprises:

and generating a third crawler task class instance object through a java reflection mechanism according to the full limit name of the third crawler task class defined in the crawler task code, and calling a page storage method of the third crawler task class instance object to store the page analysis result of the website page corresponding to the initial url.

8. The method of claim 7,

the website information to be captured comprises a data identifier generation rule and a target storage system, wherein the data identifier generation rule comprises a field participating in data identifier generation and a method for generating a data identifier;

calling a page storage method of a third crawler task instance object to store a page analysis result of a website page corresponding to the initial url, wherein the page storage method comprises the following steps:

determining the field content of each field participating in the generation of the data identification in the page analysis result;

determining a data identifier corresponding to the page analysis result according to the field content of each field participating in generating the data identifier and the method for generating the data identifier;

and calling a page storage method of a third crawler task instance object, and storing the page analysis result and the corresponding data identifier thereof to a target storage system.

9. The method of claim 8,

the fields participating in generating the data identification comprise: url field, tilt field, pubTime field;

the method for generating the data identifier comprises the following steps: and splicing the field contents of the fields participating in the generation of the data identification according to a preset sequence to generate an MD5 value corresponding to the spliced character string, and taking the MD5 value as the generated data identification.

10. A crawler apparatus, the apparatus comprising: the system comprises a task scheduling module, a page capturing module, a page analyzing module, a data storage module, a message starting module and a code publishing module;

the code publishing module is used for compiling the crawler task code corresponding to the website to be grabbed and then publishing the crawler task code to a project path of the crawler system;

the message starting module is used for sending the information of the websites to be captured to a message queue maintained by the task scheduling module; the website information to be grabbed comprises an initial url and a full-limited name of a crawler task class defined in the crawler task code;

the page grabbing module is used for acquiring website information to be grabbed from a message queue maintained by the task scheduling module, and grabbing a website page corresponding to an initial url in the website information to be grabbed according to a full limited name of a crawler task class defined in the crawler task code;

the page analysis module is used for acquiring website information to be captured from a message queue maintained by the task scheduling module, and performing page analysis on a website page corresponding to the initial url according to a full qualified name of a crawler task class defined in the crawler task code;

and the data storage module is used for acquiring the information of the website to be captured from the message queue maintained by the task scheduling module, and storing the page analysis result of the website page corresponding to the initial url by the page analysis module according to the full qualified name of the crawler task class defined in the crawler task code.

11. An electronic device, comprising: the system comprises at least one processor and a memory connected with the at least one processor through a bus; the memory stores one or more computer programs executable by the at least one processor; characterized in that the at least one processor, when executing the one or more computer programs, implements the steps in the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that the computer-readable storage medium stores one or more computer programs which, when executed by a processor, implement the steps in the method of any one of claims 1-9.

Images