CN103902386B - Multi-thread network crawler processing method based on connection proxy optimal management - Google Patents

Abstract

The invention belongs to the technical field of information processing, and specifically relates to a multi-threaded web crawler processing method based on connection agent optimization management, which first obtains a public agent server on the network, tests the network connection performance of the agent server, and obtains the optimal method according to the agent server performance. The number of threads; then manage the proxy server pool, and set an effective proxy server for each Http request; finally execute the Web page access request. The beneficial effects of the present invention are: the number of threads is obtained through calculation, which can effectively make the maximum use of resources without causing waste of resources; balance the use times of each available proxy server, and effectively prevent frequent access from being detected by the server .

Description

A multi-threaded web crawler processing method based on connection proxy optimization management

technical field

The invention relates to the technical field of information processing, in particular to a novel web page information acquisition method, in particular to a novel web crawler processing method based on the existing web crawler principle for optimal management and design of connection agents.

Background technique

With the rapid development of the network, the network has become the carrier of a large amount of information, how to effectively extract this information has become a huge challenge.

The web crawler is a very important part of the search engine system. It is responsible for collecting web pages from the Internet and collecting information. These web page information is used to build an index to provide support for the search engine. Its performance directly affects the effect of the search engine. . As the amount of network information increases geometrically, the requirements for the performance and efficiency of web crawler page collection are also getting higher and higher.

We always hope to obtain more data in a shorter period of time, but this will cause a very high load on the website, and it will also bring about problems such as increased network traffic and leakage of private data. Many websites use crawler detection technology. Analyze the web access log, and when a crawler is identified, the crawler is prohibited from using the address, and the crawler is refused to continue accessing. In order to prevent crawlers from being monitored, a large number of distributed web crawlers, camouflaged web crawlers and web crawlers using proxies have been designed for website crawler detection methods, such as Google crawlers designed by Stanford University, Disguised Spider, Internet Archive crawler. They replace the UserAgent, set the access time interval and optimize the URL access strategy, and use proxy servers, multi-threading and other methods to optimize crawlers. However, when the crawler is actually used, the following problems will be encountered: (1) There is no specific definition standard for the interval parameter, and the performance of the crawler cannot be guaranteed. When the website is very strict on crawler detection, the crawler needs a long interval, resulting in low practicality of the crawler, and shortening the interval will cause the crawler to be identified and unavailable. (2) When using a proxy server, it is necessary to optimize the crawler efficiency according to the performance of the proxy server and the number of multi-threads, but the current research does not give a specific method for how to optimize it. If an inappropriate relationship value is set, the crawler efficiency will decrease. very low.

It can be seen that on the basis of the existing crawler technology, it is very important to optimize the configuration of the crawler data acquisition thread by using a reasonable proxy connection management method to improve the performance of the crawler and prevent the crawler from being rejected by the server. The present invention provides a design method that meets this requirement.

Contents of the invention

The main purpose of the present invention is to propose a multi-threaded web crawler based on connection proxy optimization management to avoid being detected by the server for the problem that the crawler is rejected when accessing the web page. This method has a certain adaptability, and can solve the problem that the crawler is rejected when acquiring Web pages. This method makes full use of the existing network connection proxy services on the Internet, optimizes management and design of proxy connections in multi-threaded crawling, and avoids repeated use of the same client IP address to connect to the Web through the automatic selection mechanism of proxy services. server, so as to avoid being detected by the server.

The multi-threaded web crawler processing method based on connection agent optimization management proposed by the present invention mainly uses a plurality of connection agents, and proposes effective agent management strategies and parameter setting schemes, and uses multi-agents to perform data crawling on the basis of multi-thread crawlers. Pick. It first obtains the public proxy server on the network, tests the network connection performance of the proxy server, and obtains the optimal number of threads according to the performance of the proxy server; then manages the proxy server pool, and sets an effective proxy server for each Http request, and finally Execute web page access requests. in:

The number of threads M is determined according to the performance of the proxy server in the proxy pool, and the calculation formula adopted is as follows:

in, is the failure rate of the proxy server, v is the crawling speed, Serving the agents in the agent pool

The response time expectation of the device;

When managing the proxy server pool and setting an effective proxy server for each Http request, set the failure rate of the proxy server to and response time as a criterion for identifying whether a proxy is valid; after a period of time, try to use the proxy that was judged to be unusable before, so as to put the available proxy in the invalid proxy server pool into the effective proxy server pool; proxy When assigning tasks, obtain the least-used agent in the effective agent server pool for each thread, so as to distribute tasks to each agent in a balanced manner.

In the present invention, the agent task assignment is realized by adopting the priority queue with the minimum usage times. Use a linked list to store the agent, and record the number of times the agent is used within the time window. Sort the linked list in descending order of the number of times the agents are used. When inserting a proxy, use the insertion sorting method to compare the usage times of the proxy from the end of the linked list, and when the proxy is found in the linked list with a usage count not greater than that of the proxy, insert it. When a proxy needs to be provided, get the proxy at the head of the linked list and delete the proxy.

In the present invention, according to the response time of the proxy server, the average response time expectation value of the crawler is calculated ,Calculated as follows:

in, is the response time of the i-th proxy server, and N is the number of proxy servers.

In the present invention, the failure rate is calculated according to the proxy server state queue and the number of failures of the proxy server in the time window W ;Calculated as follows:

Among them, failedTimes is the number of failures of the proxy server during connection execution, and usedTimes is

The number of times the proxy server is used.

The beneficial effects of the present invention are: (1) The number of multi-threads is obtained based on the performance and number of the proxy server used. By dynamically calculating the relationship between the number of threads and the parameters of the proxy server, any available proxy server can adapt to the web crawler proposed by the present invention, and the normal use of the crawler will not be affected by the instability of the proxy server. Moreover, by calculating the number of threads, resources can be effectively utilized to the maximum without causing waste of resources. (2) A flexible method of managing proxy servers is proposed. For different visiting websites, you can choose to use a valid proxy server to access the website. (3) Propose a proxy server selection method to make the use time interval of each proxy server the longest. The crawler assigns a proxy server for each Http request for access, balances the number of times each available proxy server is used, and effectively prevents frequent access from being detected by the server.

Description of drawings

Fig. 1 is the flow chart of the present invention.

Fig. 2 is a schematic diagram of the Proxy state queue in the present invention.

detailed description

The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments.

Figure 1 is a further illustration of the process of the present invention. In the figure, the process in the virtual box A is the initial work needed to build a crawler, and it only needs to be executed once. The process in the virtual frame B is the process of the crawler crawling the web page, which needs to be executed repeatedly until the end.

(1) Obtain a proxy server and store it in the proxy server pool.

(2) Test the network connection performance of the proxy server.

(3) Create a certain number of multi-threads according to the performance of the proxy server.

(4) Convert the crawling target address started by the crawler into an Http request, and send it from the proxy server pool

Obtain a valid proxy server in , and set the Http request to be executed through the proxy server.

(5) Add the Http request to an Http request queue.

(6) The idle thread gets the Http request task from the queue Http request queue.

(7) The thread executes the Http request it gets. During execution, if a new request destination is encountered

address, create a new Http request as in step (4), and add it to the Http request queue.

(8) After the Http request is executed, get the Http response information and store the Web page information

locally.

(9) The thread returns to the idle state.

(10) Repeat steps (6) to (9) until all threads are idle and Http requests

There are no requested tasks in the queue.

In the present invention, the multi-threaded network crawler frame design based on connection agent optimization management is mainly divided into three modules: Http request module, proxy server module and crawling module. The Http request module generates an Http request to be executed according to the Http request, and puts it into the Http request queue, and at the same time obtains an effective proxy from the proxy server module, and sets the proxy to execute the Http request. Among them, the proxy server module manages the proxy server and provides effective proxy for the modules that need to use the proxy. The request tasks in the Http request queue are executed by the crawling module. The crawling module creates M parallel threads to execute tasks in the Http request queue respectively. If a new Http request is generated during execution, it will continue to be added to the Http request queue.

1) Proxy server module

1.1) Obtain a proxy server

Create a proxy server pool in the database, and add the Http proxy server to the proxy server pool for use. Available proxy servers can be found at sites on the Internet that offer proxy servers. The specific list can be obtained manually or automatically by another small crawler. The acquisition of the proxy server can be found in many websites that provide proxy servers on the Internet.

1.2) Definition of proxy server

proxy server It is defined as a six-tuple, where address is string type data, representing the address of the proxy server; port is integer data, representing the port of the proxy server; usedTimes is integer data, representing the number of times the proxy server is used; timeQueue is a Linked list, the storage value is the response time calculated in 1.3.2) , each time a new time value is added to the head of the linked list, and when the size of the linked list is greater than the maximum number of uses S in the time window W, delete the time after the Sth time in the linked list; statusQueue is also a linked list, that is, 1.3 The statusQueue defined in b.2) of .1).

1.3) Proxy servers in the management database

Proxy servers don't do their job well all the time. The present invention hopes to check whether the proxy server can be used normally, and the response time for completing an Http request is acceptable to the entire crawler system. Therefore, in order to obtain an agent that can effectively complete the access task, the present invention takes the failure rate of the agent crawling data And response time (responseTime) as a criterion to identify whether a proxy is valid.

The present invention designs an effective agent server pool and an invalid agent server pool, and stores effective agents and invalid agents respectively. Create two linked lists in the database ), where name is the table name of the linked list, set to "ValidProxyPool" and "invalidProxyPool"respectively; key is the server address of the stored proxy, and the storage format is <Address>:<Port>; see 1.2 for the definition of the proxy form), in When storing, use JSon to convert the proxy class into input stream data and store it in the database.

1.3.1) Judgment of Invalid Proxy

a) Invalid proxy definition

The present invention will calculate within a recently used time window W, when the following conditions i and ii or conditions i and iii are met at the same time, the agent is considered invalid:

i.usedTimes>Min_Used_Times

Among them, usedTimes is the number of times the proxy server is used in W, and the calculation method can be found in b.3). Min_Used_Times is the minimum number of times the proxy server is used, which can be a smaller integer, such as 10 times.

ii. failedRate>Failure_Rate

Where failedRate is the failure rate of the agent within the time window W, expressed as . failedTimes is the number of times the proxy server failed to access within W, and the calculation method can be found in b.3). Failure_Rate is the acceptable failure rate of the proxy server, which can be set to 10%.

.responseTime<Max_Resonpse_Time

Among them, responseTime is the average response time within the time window W, expressed as

.

is the response time of the i-th use of the proxy. s is the number of uses in the time window W, that is . Max_Response_Time is the longest acceptable response time of the proxy server, for example, the value can be set to 300s.

b) Calculate the failure rate of the agent

b.1) Create a new state queue with a large enough size for each agent, store the state of the agent, and define it as a two-tuple Among them, time and state respectively represent the time value at the time of use and the corresponding value state of the use result. The format is as follows:

element name time state value type Timestamp 0 or 1

in particular:

time is the moment when the idle thread obtains the request from the Http request queue in Flowchart 1.

state indicates whether the execution result is successful or not after using the proxy to execute the Http request. If its value is 0, it means the access is successful; if it is 1, it means the access failed. According to the Http response status code obtained by executing the request, when the status code is 2xx and 3xx, the access is successful; when it is other values, the access fails. If the proxy server defines the Http response status code by itself, judge whether the access is successful or not in the same way.

b.2) Agent's state queue

The agent's status queue statusQueue is also a linked list , store the status defined in b.1), each time a new status is added to the head of the linked list, and when the time value of the status in the queue is not within the time window, that is, the difference between the current time and the time value of the state If it is greater than T, delete the status.

After the agent executes the request task, it calculates the status according to the method of b.1), and adds it to the queue statusQueue.

A schematic diagram of an agent's state queue is shown in Figure 2.

b.3) Count usage and failure times

The number of times usedTimes is the number of states of the Proxy within the time window W. The number of failures failedTimes is the number of Proxy status = 1 within W.

From this, the failure rate can be obtained

1.3.2) Calculate the agent's response time expectation T

When testing the network connection performance of the proxy server, use the proxy in the proxy pool to visit the target address for s times. Here, the size of s can be slightly larger than Min_Used_Time / Failure_Rate. The response time of a certain time of Proxy For, start to execute the Http request to the end of execution. t1 is the moment when the thread obtains the request task using the agent from the Http request queue, and t2 is the moment when the request execution is completed and the thread returns to the idle state. Available:

Therefore, the available response time T is approximately the response time expectation, as follows:

which is,

1.3.3) Manage Proxy Pools

Because the proxy server may be unstable, causing it to be unavailable for a certain period of time, but it may continue to be available in the future. Or when the proxy is banned due to factors such as frequent access, but the access permission may be lifted after a period of time. Therefore, after a certain period of time, the present invention tries to use the agent judged as unusable before again, so as to put the available agents in the invalid agent server pool into the effective agent server pool.

1.4) Provide the usage method of proxy server.

The present invention hopes to use each agent in the proxy server pool as evenly as possible, reducing the problem of excessive agent load and repeated IP access and being rejected, so obtain the agent with the least number of times of use in the Proxy Pool for each thread, to balance Assign tasks to each proxy.

In the specific implementation manner, the priority queue with minimum usage times is used to ensure the usage strategy of the proxy server. The queue is represented as , use a linked list to store the proxy, and the stored value is proxy, see 1.2 for the formal definition. The priority parameter is the usedTimes attribute of the proxy, and the linked list is sorted in descending order of the times the proxy is used. When inserting a proxy, use the insertion sorting method to compare the usage times of the proxy from the end of the linked list, and when the proxy is found in the linked list with a usage count not greater than that of the proxy, insert it. When a proxy needs to be provided, get the proxy at the head of the linked list and delete the proxy. In this way, it is ensured that the agent with the least number of times of use in the agent pool is obtained each time, and the usage tasks are distributed to each agent in a balanced manner.

2) Http request module

The Http request is responsible for establishing a connection request in the crawler to obtain web page information. Its working principle is: execute Http request and return Http response result. The specific process is as follows:

For each Http request, create a corresponding Http request object. And use the proxy connection obtained in the proxy module to obtain the proxy from the effective proxy server pool, and use HttpClient to set the proxy information of the Http request.

2.1) Execute crawling tasks.

2.2) Crawl the corresponding page content according to the Http request information.

3) Crawling module

This module is responsible for creating multi-threaded crawling tasks and establishing Http request queue lists.

3.1) Create a multi-threaded crawling task

Each thread gets the crawling task from the Http request queue. After the current crawling task is executed, it continues to get the task from the queue until the thread ends. In order to speed up the crawling speed of crawlers, the present invention will establish a thread pool to allow multiple threads to crawl data in parallel.

3.1.1) Calculate the performance of the proxy server in the proxy pool to get the expected response time

Assuming that there are a total of N proxies available, the total expected response time is obtained:

is the response time of the i-th proxy server, see 1.3.2 for the calculation method). Because, as in the previous design for the use of agents, the present invention hopes to approximate as much as possible the probability that each agent uses is equal, that is:

3.1.2) Create M threads in the thread pool

According to the operating conditions of the proxy server, calculate the appropriate M to make the crawler run efficiently. The number of threads is set according to the average response time of available agents in the effective agent server pool and the expected data acquisition speed of the present invention.

For M threads, the time cost t to complete Q data requests is:

According to the response time expectation of the proxy server, the present invention can set different numbers of threads according to the required data crawling speed, so as to efficiently complete the data crawling task.

which is:

3.2) Establish a multi-threaded crawling task Http request queue list

The establishment of the Http request queue is similar to the establishment of the URL queue in the mature web crawler on the Internet. Using the first-in-first-out mode, starting from the Http request as the crawling entry, parsing the corresponding page, extracting the information set of the next address to be crawled from the page, constructing the Http request, and using HttpClient to create a new Http request, And add the Http request to the Http request queue.

As can be seen from the above-mentioned implementation process, the present invention dynamically manages the proxy server during the process of crawler obtaining Web page information, automatically adjusts the parameter relationship between the proxy server and the number of threads, and avoids the problem by automatically selecting the proxy service in multi-threaded crawling. Reuse the same client IP address to connect to the web server, thus avoiding detection by the server.

Claims (4)

1. A multithreading web crawler processing method based on connection proxy optimization management is characterized by comprising the following specific steps of firstly obtaining a public proxy server on a network, testing the network connection performance of the proxy server, and obtaining the optimal thread number according to the performance of the proxy server; then, managing a proxy server pool, and setting an effective proxy server for each Http request; finally executing the Web page access request; wherein:

the thread number M is determined according to the performance of the proxy servers in the proxy server pool, and the calculation formula is as follows:

here, ,the failure rate of the proxy server, v the crawling speed,the expected value of the response time of the proxy server in the proxy server pool is obtained;

managing the proxy server pool, and setting a valid proxy server for each Http request, determining the failure rate of the proxy serverAnd response time as a criterion for identifying whether an agent is valid; after the interval, trying again to use the agent which is judged to be unusable before, thereby putting the usable agent in the invalid agent server pool into the valid agent server pool; when the agent task is distributed, the agent with the least use times in the effective agent server pool is obtained for each thread, so that the task is distributed to each agent in a balanced manner; wherein: the invalidation agent is defined as follows:

calculating that within a time window W of recent use, the agent is considered invalid when the following conditions i and ii or conditions i and iii are satisfied simultaneously;

i.usedTimes>Min_Used_Times；

ii.failedRate>Failure_Rate；

responsetime < Max _ Response _ Time, where:

usedTimes is the number of Times the proxy server is Used in W, Min _ Used _ Times is the set minimum number of Times the proxy server is Used, failedRate is the Failure Rate of the proxy in Time window W, Failure _ Rate is the Failure Rate of the acceptable proxy server, responseTime is the average Response Time in Time window W, and Max _ Response _ Time is the longest Response Time of the acceptable proxy server.

2. The multithreading web crawler processing method based on connection broker optimization management of claim 1, wherein: and realizing agent task allocation by adopting a priority queue with minimum use times.

3. The multithreading web crawler processing method based on connection broker optimization management of claim 1, wherein: response time expectation of the proxy serverThe calculation formula of (a) is as follows:

wherein,the response time of the ith proxy server is shown, and N is the number of the proxy servers.

4. The multithreading web crawler processing method based on connection broker optimization management of claim 1, wherein: failure rate of the proxy serverAccording to the state queue of the proxy server and the failure times of the proxy server in the time window w, the calculation formula is as follows:

the failedTimes are failure times of the proxy server during connection execution, and the usedTimes are usage times of the proxy server.