CN111651656B - Method and system for dynamic webpage crawler based on agent mode - Google Patents

Abstract

The invention discloses a method and a system for dynamic webpage crawler based on a labor mode, wherein the method comprises the following steps: receiving service information, configuring crawler parameters, evaluating the service and preparing the service; allocating system resources and initiating a plurality of independent process business crawlers; crawling the original URL of the dynamic webpage by adopting a simulated browser mode, and returning the URL of the target static data content; reviewing validity and non-repeatability of the URL, constructing a production task message list by using the crawl tasks after review, and initiating a plurality of threads of production crawlers; crawling a static URL page by adopting an automatic program mode, and returning target data and an attachment file; processing and storing the returned content; and exporting the data. The invention respectively constructs a business crawler and a production crawler, adopts different crawling strategies for the dynamic webpage and the static content based on a substitute mode, utilizes system resources to the maximum extent and realizes large-scale and quick crawling of dynamic webpage data.

Description

A method and system for dynamic webpage crawler based on foundry mode

technical field

The invention relates to the technical fields of Internet information retrieval and search engines, in particular to a dynamic web page crawling method and system based on an OEM mode.

Background technique

Web crawlers are an important part of Internet search engines, and are mainly used to crawl data in web pages on the Internet and build indexes for search engines. Whether the amount of crawling is large enough determines whether the content of the search engine is rich and whether the crawling is immediate, directly affects the overall effect of the search engine. In the context of big data, web crawlers are also widely used to capture network data such as Internet public opinion, commodity transactions, cultural and sports entertainment, and provide massive basic data for further data mining and data analysis.

The working principle of a general web crawler is to obtain the HTML data of the web page by accessing the URL of the target web page, then parse the DOM nodes in the HTML, extract the target data or the URL link of the data, save it to the database or, and then use depth-first or breadth-first Wait for the strategy to continue to crawl data on more web pages. Because web crawlers will more or less interfere with the target website or for reasons of data protection, many websites will adopt some anti-crawler means. In addition, due to their business functions, some websites will not display all the information when the user initially opens the page, but need to click a button or slide the scroll bar after manual operations, and then pass Ajax The method is dynamically loaded in. For the crawling of dynamic web pages, the prior art method adopts the mode of simulating browsers (such as Selenium, PhantomJS, etc.), and simulates the behavior of mouse and keyboard through programs in places where manual operation is required, so as to trigger web pages to dynamically load new content. the goal of. The biggest disadvantage of this method is low efficiency and simple task scheduling, which cannot meet the needs of large-scale crawling tasks.

Contents of the invention

The technical problem to be solved by the present invention is that the existing crawler method has low efficiency, simple task scheduling, and cannot meet the needs of large-scale crawling tasks. The problem of completing the crawling task.

The present invention realizes through following technical scheme:

A dynamic webpage crawling method based on an OEM mode, comprising the following steps: S1: receiving business information input by a user, configuring crawler business parameters, performing business evaluation, and making preparations; S2: according to the business information, specifying Time allocation of system resources, launch business crawlers with multiple independent processes; S3: The business crawler adopts the simulated browser mode to crawl the original URL of the dynamic web page, and return the URL of the target static data content; S4: Review the The effectiveness and non-repetition of the URL, and for the crawling tasks that have passed the review, construct a production task message list, and initiate multiple multi-threaded production crawlers on the distributed server; S5: the production crawler adopts an automated program mode, Crawl the URL page containing static content, and return the target data field and the attachment file; S6: perform preprocessing on the target data field, the preprocessed target data field and the attachment file form business data, and store the Business data; S7: Export the business data and feed it back to the user.

The present invention adopts different crawling strategies for dynamic webpages and static content, exerts their respective advantages in a targeted manner, and maximizes the use of system resources; based on the foundry model, the method is operated according to the business function, and the method can be carried out according to the specific situation of the business The corresponding adjustment and configuration of the system, the orderly scheduling and mutual cooperation between the steps, has a high degree of flexibility, and can realize large-scale and fast crawling of dynamic web pages. The method of the present invention is based on the foundry mode, and adopts a business crawler that simulates browser operation, independent process, and synchronous mode for dynamic web pages; after crawling to the link URL of the static content of the target data, the delivery efficiency is higher, and automatic programs, multiple Production crawlers in threaded and asynchronous mode crawl in parallel. As a result of business evaluation, if the amount of data is huge and the parallelism of the production crawler is high, it will be stored in a sub-database and sub-table manner to improve access efficiency.

Further, the business information includes business ID, business description, starting URL, login information, execution strategy, anti-crawling measures, crawling data fields and positioning and exporting data formats.

Further, the login information includes account number, password and CA certificate, the execution strategy is one-time timing incremental crawling, and the anti-crawling measures include IP proxy, browser header and CSS offset. If the target URL is set with anti-crawling measures, when the production crawler is started, it will be crawled according to the anti-crawling measures.

Further, the preparatory work in step S1 includes: business parameter completeness check: ensuring the integrity of the business information; ensuring the integrity of the business-related information input by the user, and the system can execute the crawling task according to the relevant parameters. Access pass preparation: According to the business information, log in to the target website, obtain and save cookies information, verify the CA certificate; ensure that the account is successfully logged in. Small task attempt: according to the preparation of the starting URL and the access pass, construct a small crawling task, and analyze the character encoding, data volume, access duration and crawling success rate of the target website through execution; for the next step Arranging formal crawling tasks provides resource arrangement reference information. Design data fields and standardization requirements: According to the business information, design the field names, field formats, field lengths and standardization requirements of the target data. Create databases, data tables, and storage space for attachments. Establish persistent storage related resources for this business.

Further, the business evaluation in step S1 includes evaluation of server hardware, software, storage and network bandwidth required by the business. Based on the current operating status and resource conditions of the system, judge whether the business can be undertaken immediately; if it can be undertaken, the business preparation will be started; if it cannot be undertaken, it will be fed back to the user, and it will not be executed immediately.

Furthermore, if the amount of data is huge and the production crawler has a high degree of parallelism, it will be stored in a sub-database and sub-table manner. To improve access efficiency.

Further, in the step S3, the simulated browser mode includes simulated browser mouse click, scroll bar sliding, keyboard input and copy and paste.

Further, the step S4 includes the following sub-steps: S01: Check the validity of the URL through the legality rules; S02: Map the URL to the Key of the HashMap through the Hash algorithm, and use the Bloom filter to check whether the URL is repeated; if repeated , it is discarded and logged.

Further, in the step S6, the target data field is preprocessed, and the details are as follows: data integrity check: whether the specified field to be crawled is complete, and whether the data content meets the requirements; for data that cannot meet the requirements It can be corrected, marked or discarded according to the severity; field standardization processing: whether the data format and data type of the field conform to the setting; individual conversion of the data that cannot meet the requirements; attachment file transfer: virus detection of crawled attachment files After scanning, reassign a non-repeatable file name with a time stamp, and then transfer it to a fixed location for storage, and add a new data field to record the path and file name of the attached file.

Another implementation of the present invention is a dynamic web crawler system based on OEM mode, including: a business interface module: as a user interface related to business, it receives business information input by users, configures parameters related to crawler business, and performs business Evaluate and make preparations; the business information includes: business ID, business description, starting URL, login information, execution strategy, anti-crawling measures, crawling data fields and positioning, and final export data format; the preparation The work includes completeness inspection of business parameters, preparation of access passes, small task attempts, design of data fields and standardization requirements, establishment of databases, establishment of data tables and establishment of storage space for attachments; the business assessment includes server hardware, software, Evaluation of storage and network bandwidth; business scheduling module: allocate system resources at a specified time according to business-related information, and initiate multiple independent processes of business crawlers; business crawler: use simulated browser mode to crawl the original URL of dynamic web pages , and return the URL of the target static data content; production scheduling module: receive the crawling tasks returned by the business crawler, review the validity and non-repetition of the URL, and construct a production task message list for the crawling tasks that have passed the review, in the distribution Initiate multiple multi-threaded production crawlers on a cluster server; production crawlers: use automated program mode to crawl URL pages with static content, and return target data and attachment files; storage module: receive target data returned by production crawlers and attachment files, completeness and standardization processing are carried out to the target data, and the processed target data and attachment files are stored in the database to form business data; export module: export business data according to the final export data format, feedback to the user.

Evenly distribute crawling tasks to distributed servers through load balancing, and start the production crawler to crawl the target data in a multi-threaded and asynchronous manner; the production crawler uses an automated program to access the target URL, obtains the target data according to the location and sends it to the storage module , and at the same time feed back the task execution status to the production scheduling module. A number of independent processes initiated by the business scheduling module according to business requirements and related information, and business crawlers simulating browser mode, execute crawling of dynamic web pages. Based on the foundry model, the system adopts a business crawler that simulates browser operation, independent process, and synchronous mode for dynamic web pages; after crawling to the link URL of the static content of the target data, the delivery efficiency is higher, using automated programs, multi-threading, The production crawler in asynchronous mode crawls in parallel. As a result of business evaluation, if the amount of data is huge and the parallelism of the production crawler is high, it will be stored in a sub-database and sub-table manner to improve access efficiency. The export module exports the crawled data from the database and attachment files according to the requirements of the format and form of the exported data in the business parameters, and generates statistical description information of the data at the same time, and delivers it to the user after packaging.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

The present invention adopts different crawling strategies for dynamic webpages and static content, exerts their respective advantages in a targeted manner, and maximizes the use of system resources; The specific business conditions are configured and transformed, and the modules are scheduled and coordinated with each other in an orderly manner. It has a high degree of flexibility and can realize large-scale and fast crawling of dynamic web pages.

Description of drawings

The drawings described here are used to provide a further understanding of the embodiments of the present invention, constitute a part of the application, and do not limit the embodiments of the present invention. In the attached picture:

Fig. 1 is a system structure diagram of the present invention;

Fig. 2 is a method architecture diagram of the present invention;

Fig. 3 is a functional diagram of the service interface module of the present invention;

Fig. 4 shows the corresponding relationship between the business crawler and the production crawler and the content of the task list in the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the examples and accompanying drawings. As a limitation of the present invention.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one of ordinary skill in the art that these specific details need not be employed to practice the present invention. In other instances, well-known structures, circuits, materials, or methods have not been described in detail in order to avoid obscuring the present invention.

Throughout this specification, reference to "one embodiment," "an embodiment," "an example," or "example" means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in the present invention. In at least one embodiment. Thus, appearances of the phrases "one embodiment," "an embodiment," "an example," or "example" in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, particular features, structures or characteristics may be combined in any suitable combination and/or subcombination in one or more embodiments or examples. Furthermore, those of ordinary skill in the art will appreciate that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Example

This embodiment is a dynamic webpage crawler system based on the foundry model. As shown in Figure 1, the system structure is divided into a business interface module, a business scheduling module, a business crawler, a production scheduling module, a production crawler, a storage module and an export module. parts. The main functions of each part are as follows:

(1) Business interface module: As a business-related user interface, it receives user input, configures crawler business-related parameters, conducts business evaluation, and makes relevant preparations;

(2) Business scheduling module: according to business-related information, allocate system resources at a specified time, and initiate several business crawlers of independent processes;

(3) Business crawler: use the simulated browser mode to crawl the original URL of the dynamic webpage, and return the URL of the target static data content;

(4) Production scheduling module: receive crawling tasks returned by business crawlers, review the validity and non-repetition of URLs, construct a production task message list for the crawling tasks that have passed the review, and initiate several multiple crawling tasks on the distributed cluster server. threaded production crawler;

(5) Production crawler: adopt automatic program mode to crawl URL pages containing static content, and return target data fields and attachment files;

(6) Storage module: Receive the data and attachments returned by the production crawler, complete and standardize the data fields, dump the attachment files, and store all the data through the database;

(7) Export module: export business data according to the requirements of the business on the final export data format, and feed back to the user.

As shown in Figure 2, the relationship between the various modules of the system corresponds to the relevant links of industrial OEM production. Within each module, configuration and transformation can be carried out according to the specific business conditions. The orderly scheduling and mutual cooperation between the modules has a high degree of flexibility. In the crawling production process, the business crawler that simulates browser operation, independent process, and synchronous mode is used for dynamic web pages; after crawling to the link URL of the static content of the target data, the delivery efficiency is higher, using automated programs and multi-threading , Parallel crawling by production crawlers in asynchronous mode.

In the specific implementation process, hardware servers can be deployed according to the actual workload of the business. If the business volume is large, hardware servers can be deployed for each module, among which the business crawler and production crawler can be deployed as a distributed cluster system, and the storage module can deploy multiple servers according to each database; if the business volume is not large, the business interface can be The module, business scheduling module and business crawler are combined on one server, and the production scheduling module, production crawler, storage module and export module are deployed on one server; if the business volume is really small, all modules can even be deployed on one server on the server.

As shown in Figure 2, the system first receives user-input crawler business-related information as parameters for subsequent production scheduling. The specific content includes: business ID, business description, starting URL, login information (account, password, CA certificate, etc.), Strategy (one-time crawling, timing incremental crawling), anti-crawling measures (IP proxy, browser header, CSS offset, etc.), crawling data fields and positioning, and final export data format and other business-related information; all information After being edited by the user, it is saved in a Json format file, and the system obtains business parameter information by reading the Json file.

The business interface module is responsible for interacting with users, receiving relevant parameters input by users, and performing review, evaluation and business preparation. The main functions are shown in Figure 3.

Among them, the business evaluation in the business interface module includes the evaluation of server hardware, software, storage, network bandwidth and other resources required by the business; combined with the current system operation status and resource conditions, it is judged whether the business can be undertaken immediately; if the business can be undertaken Then start business preparations, if it cannot be undertaken, it will be fed back to the user, and it will not be implemented immediately.

Relevant preparations in the business interface module include:

(1) Completeness check of business parameters: the business-related information entered by the user is complete, and the system can perform crawling tasks based on relevant parameters;

(2) Access pass preparation: log in to the target website according to the login information, obtain and save the cookie information, or verify the CA certificate to ensure that the account can be successfully logged in; some websites directly obtain the individual key name of the cookie after login and access the network with the cookie The key names of the cookies are different. Therefore, after obtaining the cookies for the first time, you need to verify whether you can log in successfully. If not, you need to further check the key names of the cookies;

(3) Small task attempt: According to the starting URL and access pass in the business parameters, construct a small crawling task, analyze the character encoding, data volume, access time, crawling success rate and other information of the target webpage through the execution status, and provide The next step is to arrange formal crawling tasks to provide resource arrangement reference information;

(4) Design data fields and standardization requirements: According to the crawling target and business requirements, design the field name, field format, field length and standardization requirements of the target data;

(5) Establish persistent storage related resources such as database, data table, and attachment storage space for the business.

As shown in Figure 2, after the small task is successfully crawled and the storage specification and space preparation are completed, the production scheduling stage can be entered. First, the task is decomposed, and several independent processes and simulations are initiated by the business scheduling module based on business requirements and related information. A business crawler in browser mode, which crawls dynamic web pages.

The execution strategy of scheduling business crawlers is divided into two methods: one-time crawling and timing incremental crawling. One-time crawling is to execute all crawling tasks at once based on information such as the URL of the target webpage, keywords, update time, maximum number of pages, and maximum depth. Timing incremental crawling is to start the crawling task at the specified crawling time, first find the location of the latest crawling from the database or log records, and then crawl the newly added content. Crawling strategy methods can be set according to business requirements.

If you use the timing incremental crawling method, pay attention to check the validity of the access pass again before crawling to prevent the interval between crawling tasks from exceeding the user login expiration time set by the target website. If your access pass has expired, you will need to obtain it again.

The business crawler can simulate actions such as browser mouse click, scroll bar sliding, keyboard input, copy and paste, etc., and send the link URL of the target page to the production scheduling module in a synchronous manner, and at the same time feed back the task execution status to the business scheduling module. For specific implementation, Selenium, PhantomJS, etc. can be used for development and implementation.

After receiving the crawling tasks returned by the business crawler, the production scheduling module forms a task message queue and enters the review process in turn. The specific steps are:

First, the validity of the URL is checked through legality rules;

Then, use the Hash algorithm to map the URL to the Key of the HashMap, and use the Bloom filter to check whether the URL of the business is duplicated; if there is, it will be discarded and logged; if not, it will be initiated on the distributed cluster server Produce crawlers and perform crawling of static web pages.

The production scheduling module evenly distributes crawling tasks to distributed servers through load balancing, and starts the production crawler to crawl the target data in a multi-threaded and asynchronous manner; the production crawler uses an automated program to access the target URL, obtains the target data according to the location and sends it to the storage module, and at the same time feed back the task execution to the production scheduling module. It can be developed and implemented by using Scrapy, Urllib, Requests, etc. during specific implementation.

If the target website has anti-crawling measures, crawl according to the anti-crawling strategy when the production crawler is enabled. In the specific implementation, the conventional anti-crawling functions such as IP proxy, access frequency adjustment, User-Agent customization, etc. can be realized first. If the target website also adopts other anti-crawling strategies, it can be expanded according to the specific situation. Anti-reptile strategy.

There is generally a 1:N correspondence between business crawlers and production crawlers, and the contents of the specific task list are shown in Figure 4. The production crawler should be able to trace the source according to the business crawler ID. When encountering crawling exceptions, you can find the corresponding business crawler from the log and analyze the execution status of other crawlers sent by the business crawler. For the positioning of data fields on the page, various methods can be used in the specific implementation, such as find, CSS, XPath, etc. For the convenience of writing and searching, the XPath method is generally used.

As shown in Figure 2, the storage module receives the data and attachments captured by the production crawler and puts them in the cache, preprocesses the data and then transfers it to the database for persistent storage. Data preprocessing includes:

(1) Data integrity check: Specify whether the fields to be crawled are complete and whether the data content meets the requirements; the data that cannot meet the requirements can be corrected, marked or discarded according to the severity;

(2) Field standardization processing: whether the data format and data type of the field conform to the setting; convert the data that cannot meet the requirements separately (for example, Sina Weibo and some forum websites describe the time of new posting as ** minutes Before, or use "today" and "yesterday" to represent the date, which needs to be converted to a standard date and time format);

(3) Attachment file transfer: After scanning the crawled attachment files for viruses, reassign the file names with time stamps that are not easy to be repeated, and then transfer them to a fixed location for storage, and add a new data field for recording attachments Information such as the path and file name of the file.

During the persistent storage process of the database, according to the results of the previous business evaluation, if the amount of data is huge and the parallelism of the production crawler is high, the method of sub-database and sub-table storage is used to improve access efficiency. In actual implementation, it is generally divided horizontally. The storage location of a record can be selected by modulus, that is, idmodn, and then stored in the database; Resources need to be expanded, and databases and servers can be added.

The export module exports the crawled data from the database and attachment files according to the requirements of the format and form of the exported data in the business parameters, and generates statistical description information of the data at the same time, and delivers it to the user after packaging. During specific implementation, a user interface can be developed, input business ID or related information, check the current business execution status, data size and other information, and then export it by the system administrator or authorized account according to the established rules.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Protection scope, within the spirit and principles of the present invention, any modification, equivalent replacement, improvement, etc., shall be included in the protection scope of the present invention.

Claims (6)

1. A dynamic webpage crawler method based on a proxy mode is characterized by comprising the following steps:

s1: receiving service information input by a user, configuring crawler service parameters, performing service evaluation, and preparing;

s2: according to the service information, system resources are distributed at a specified time, and a plurality of service crawlers with independent processes are initiated;

s3: the business crawler crawls the original URL of the dynamic webpage by adopting a simulated browser mode and returns the URL of the target static data content;

s4: checking the validity and non-repeatability of the URL, constructing a production task message list for the crawl tasks passing the checking, and initiating a plurality of multi-thread production crawlers on a distributed server;

s5: the production crawler crawls URL pages containing static content by adopting an automatic program mode, and returns target data fields and attachment files;

s6: preprocessing the target data field, forming service data by the preprocessed target data field and the attachment file, and storing the service data;

s7: exporting the service data and feeding back the service data to a user;

the service information comprises a service ID, a service description, an initial URL, login data, an execution strategy, anti-crawling measures, a crawling data field and a positioning and exporting data format;

the login data comprises an account number, a password and a CA certificate, the execution strategy is one-time timed incremental crawling, and the anti-crawling measures comprise an IP agent, a browser head and CSS offset;

wherein the preparation work in the step S1 includes:

checking the completeness of the service parameters: ensuring the integrity of the service information;

preparing an access pass: logging in a target website according to the service information, acquiring and storing Cookies information, and verifying the CA certificate;

and (3) small task trying: constructing a small-sized crawling task according to the initial URL and the access pass preparation, and analyzing character codes, data volume, access duration and crawling success rate of the target website through execution conditions;

design data fields and normalization requirements: according to the service information, designing a field name, a field format, a field length and a standardization requirement of target data;

establishing a database, a data table and an accessory storage space;

the service evaluation in step S1 includes evaluation of server hardware, software, storage and network bandwidth required by the service.

2. The dynamic web page crawler method based on the agent model as recited in claim 1, wherein if the data size is huge and the parallel degree of the production crawler is high, the data is stored in a database-based and table-based manner.

3. The method according to claim 1, wherein the browser-simulating mode comprises mouse click, scroll bar sliding, keyboard input and copy and paste in step S3.

4. The agent mode-based dynamic web crawler method according to claim 1, wherein the step S4 comprises the following sub-steps:

s01: checking the validity of the URL through a validity rule;

s02: mapping the URL to a Key of the HashMap by a Hash algorithm, and adopting a bloom filter to check whether the URL is repeated; and if the data is repeated, discarding and logging.

5. The method according to claim 1, wherein in step S6, the target data field is preprocessed, and details thereof are as follows:

data integrity checking: whether fields to be crawled are complete or not and whether data content meets requirements or not are specified; the data which can not meet the requirement can be corrected, marked or discarded according to the severity;

field normalization processing: whether the data format and the data type of the field accord with the setting or not; carrying out individual conversion on data which cannot meet the requirements;

and (3) unloading the accessory file: after virus scanning is carried out on the crawled attachment file, the file name which contains the timestamp and is not easy to repeat is redistributed, then the file name is stored in a fixed position, and a data field is newly added for recording the path and the file name of the attachment file.

6. A dynamic web crawler system based on a proxy mode, comprising:

a service interface module: as a user interface related to the service, receiving service information input by a user, configuring parameters related to the crawler service, performing service evaluation, and making preparation work; the service information includes: service ID, service description, initial URL, login data, execution strategy, anti-reverse climbing measure, climbing data field and positioning and finally exporting data format; the preparation work comprises the steps of checking the completeness of service parameters, preparing access passes, trying small tasks, designing data fields and standardization requirements, establishing a database, establishing a data table and establishing an accessory storage space; the service evaluation comprises the evaluation of server hardware, software, storage and network bandwidth required by the service;

a service scheduling module: according to the service related information, system resources are distributed at the appointed time, and a plurality of service crawlers with independent processes are initiated;

and (3) service crawler: crawling the original URL of the dynamic webpage by adopting a simulated browser mode, and returning the URL of the target static data content;

a production scheduling module: receiving a crawling task returned by the service crawler, checking the validity and non-repeatability of the URL, constructing a production task message list for the crawling task passing the checking, and initiating a plurality of multi-threaded production crawlers on the distributed cluster server;

producing the reptiles: crawling URL pages containing static content by adopting an automatic program mode, and returning target data and attachment files;

a storage module: receiving target data and an attachment file returned by the production crawler, performing integrity and standardization processing on the target data, and storing the processed target data and the processed attachment file into a database to form service data;

a derivation module: business data are exported according to the final export data format and fed back to a user;

wherein, the service parameter completeness check means: ensuring the integrity of the service information;

the access pass preparation is that: logging in a target website according to the service information, acquiring and storing Cookies information, and verifying a CA certificate;

the small task attempt refers to: constructing a small-sized crawling task according to the initial URL and the access pass preparation, and analyzing character codes, data volume, access duration and crawling success rate of the target website through execution conditions;

the design data field and the normalization requirement refer to: and designing the field name, the field format, the field length and the standardization requirement of the target data according to the service information.

Images