CN110390043A - Method, device, terminal and storage medium for crawling web mailbox data - Google Patents

Abstract

The present invention relates to deep layer net page crawler technology fields, more particularly to a kind of crawling method, device, terminal and the storage medium of webpage mailbox data, comprising: after mailbox homepage loads successfully, call the call back function of browser, wherein, the call back function includes the script file of injection；Specified search information is obtained, and the mail data of the mailbox homepage is crawled by the script that crawls of the script file, obtains corresponding with described search information crawling data；After the completion of crawling operation, by it is described crawl data and be uploaded to server parse；The parsing result that the server returns is received, and the parsing result is shown, wherein the parsing result includes the target data to match with described search information；This solution avoids server-sides, and because repeatedly crawling data, shielded phenomenon occurs, while saving server-side and crawling the resource consumed in data procedures.

Description

Method, device, terminal and storage medium for crawling web mailbox data

technical field

The present invention relates to the technical field of deep web crawlers, in particular to a method, device, terminal and storage medium for crawling web mailbox data.

Background technique

With the continuous development of computers and the Internet, the information received by users is becoming more and more complicated, and it is more inconvenient to find other information through the received information; at present, when users query the information they need, they crawl webpage information through the server And after parsing, load it to the relevant page for display.

For example, when a user logs in to the mailbox and obtains relevant information through the email, the user account and password are first crawled, and the server uses the user account and password to log in, and then crawls and analyzes the content of the email in the user mailbox. For details, please refer to the attached Figure 1; however, during this operation, the server often accesses the user’s mailbox too frequently due to repeated logins on the mailbox’s home page, which takes up a lot of computing resources, and the server’s ID is also easily blocked by the website, making it impossible to perform Follow up.

Contents of the invention

The purpose of the present invention is to at least solve one of the above-mentioned technical defects, especially the technical defect in the prior art that the server crawls webpage information too frequently, which is not only easily blocked by the website, but also occupies a large amount of computing resources.

The present invention provides a crawling method for web mailbox data, comprising the following steps:

After the main page of the mailbox is successfully loaded, the callback function of the browser is called, wherein the callback function includes an injected script file;

Obtain specified search information, and crawl the mail data of the main page of the mailbox through the crawling script of the script file to obtain crawling data corresponding to the search information;

After the crawling operation is completed, upload the crawled data to the server for analysis;

receiving the parsing result returned by the server, and displaying the parsing result, wherein the parsing result includes target data matching the search information.

In one of the embodiments, after the main page of the mailbox is successfully loaded, before the step of calling the callback function of the browser, it also includes:

Obtain the script file from the specified directory according to the version information file of the client;

Inject the script file into the callback function of the browser.

In one of the embodiments, before the step of obtaining the script file from the specified directory according to the version information file of the client, it also includes:

requesting a communication interface, and obtaining an updated version information file through the communication interface;

comparing the updated version information file with the original version information file;

It is determined whether the original version information file needs to be updated according to the comparison result.

In one of the embodiments, after the step of determining whether the original version information file needs to be updated according to the comparison result, it also includes:

When the original version information file needs to be updated, download and save the updated version information file;

comparing the check value of the updated version information file with the check value of the communication interface;

Determine whether the updated version information file is downloaded correctly according to the comparison result.

In one of the embodiments, the step of obtaining the script file from the specified directory according to the version information file of the client includes:

Searching for a configuration file by specifying a directory, wherein the configuration file includes a plurality of script files and configuration data corresponding to the script files;

When the script file exists, obtain configuration data corresponding to the script file, and compare the check value of the configuration data with the check value of the script file;

Acquire the script file according to the comparison result.

In one of the embodiments, the search information includes credit card billing information;

Obtain specified search information, and crawl the mail data of the main page of the mailbox through the crawling script of the script file, and obtain the steps of crawling data corresponding to the search information, including:

Execute the script file according to the specified credit card bill information, wherein the script file includes a crawling script;

Using the crawling script to crawl email data related to the credit card bill information in the main page of the mailbox;

Count the email data crawled by the crawling script multiple times, and obtain the crawled data corresponding to the credit card bill information according to the statistical results.

In one of the embodiments, the step of receiving the parsing result returned by the server and displaying the parsing result, wherein the parsing result includes target data matching the search information, includes:

Obtaining an analysis result returned by the server, wherein the analysis result includes bill data matching the credit card bill information;

The billing data is displayed on the main page of the mailbox.

The present invention also provides a crawling device for webpage mailbox data, which includes:

The calling module is used to call the callback function of the browser when the main page of the mailbox is loaded successfully, wherein the callback function includes the injected script file;

The crawling module is used to obtain specified search information, and crawls the mail data of the main page of the mailbox through the crawling script of the script file to obtain crawling data corresponding to the search information;

A data transmission module, configured to upload the crawled data to a server for parsing after the crawling operation is completed;

The information display module is configured to receive the parsing result returned by the server and display the parsing result, wherein the parsing result includes target data matching the search information.

The present invention also provides a terminal, which is characterized in that it includes a memory and a processor, the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes The steps in the crawling method for web mailbox data as described in any one of the above-mentioned embodiments.

The present invention also provides a storage medium, where computer-readable instructions are stored in the storage medium, and when the computer-readable instructions are executed by one or more processors, one or more processors execute the method described in the above-mentioned embodiments. The step of the crawling method of web mailbox data described in any one.

The crawling method, device, terminal and storage medium of the above-mentioned web mailbox data, when the main page of the mailbox is successfully loaded, call the callback function of the browser, wherein the callback function includes the injected script file; obtain the specified search information, and Crawl the mail data of the main page of the mailbox through the crawling script of the script file to obtain the crawling data corresponding to the search information; when the crawling operation is completed, upload the crawling data to the server Parsing; receiving the parsing result returned by the server, and displaying the parsing result, wherein the parsing result includes target data matching the search information.

In this solution, the script file in the callback function is injected in advance, and after the main page of the mailbox is successfully loaded, the callback function in the browser is called. At this time, the injected script file starts to run, and the mail data in the main page of the mailbox is crawled. , in order to obtain the crawled data corresponding to the specified search information. After the crawling operation is completed, all the crawled data will be uploaded to the server for analysis. In this way, the analysis result returned by the server is the specified search information in this solution. Matching target data; this solution can directly use script files to extract user-related information on the client side, without completely crawling relevant content through the server, and avoiding the phenomenon that the server is blocked due to multiple crawling data. At the same time, The script file is injected into the callback function of the browser instead of the server to perform the crawling operation, which saves the resources consumed by the server in the process of crawling data.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a schematic structural diagram of the background technical solution of the present invention;

Fig. 2 is the application environment diagram of the scheme of the embodiment of the present invention;

Fig. 3 is a flowchart of a crawling method for web mailbox data of an embodiment;

Fig. 4 is a schematic diagram of webpage mailbox data interaction of the present invention;

Fig. 5 is a schematic structural diagram of a crawling device for web mailbox data according to an embodiment;

Fig. 6 is a partial structural block diagram of a mobile phone related to a terminal provided by an embodiment.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

Those skilled in the art will understand that unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the word "comprising" used in the description of the present invention refers to the presence of said features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components, and/or combinations thereof.

Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this invention belongs. It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have meanings consistent with their meaning in the context of the prior art, and unless specifically defined as herein, are not intended to be idealized or overly Formal meaning to explain.

Referring to FIG. 2, FIG. 2 is an application environment diagram of an embodiment of the present invention; in this embodiment, the technical solution of the present invention can be implemented based on a terminal device 110. As shown in FIG. Install the client in the form, the user browses relevant pages by logging in the client, and the client crawls the user-related information through the script file and transmits it to the server 120 through the communication network to realize related functions; in the embodiment of the present invention, the user logs in through the terminal device 110 Client, the script file crawls the behavior information of the user on the client, and obtains matching target data according to the behavior information, and transmits the target data to the server 120, and the server 120 parses the crawled target data to return the specific The data information; the terminal device 110 here may be a smart phone, a tablet computer, a PC, etc., but is not limited thereto, and the server 120 here refers to a server device that implements various background functions.

In one embodiment, as shown in FIG. 3 , FIG. 3 is a flow chart of a method for crawling webpage mailbox data in one embodiment. In this embodiment, a method for crawling webpage mailbox data is proposed, which may specifically include the following steps :

S110: After the main page of the mailbox is successfully loaded, call a callback function of the browser, wherein the callback function includes an injected script file.

In this step, after the user successfully logs in, the page masking animation is turned on on the main mailbox main page to display the loading effect of "crawling and loading". The main purpose is to cover the main mailbox main page and prevent user operations from interfering with automatic crawling.

The purpose of using the callback function to access the email login webpage here is that the callback function is a native control in the client system code, which can be understood as a simple browser in the system. When accessing the email login webpage, it will be called when the webpage is loaded successfully. At this point, the script file has been injected into the mailbox login webpage. When the callback function is called, the js code after the script injection will be in the running state, and the user's operation will be detected in real time.

S120: Obtain specified search information, and crawl the mail data of the main page of the mailbox through the crawling script of the script file to obtain crawled data corresponding to the search information.

In this step, after the main page of the mailbox is successfully loaded through step S110, the callback function of the browser is called, wherein, after the callback function includes the injected script file, the specified search information is obtained, and crawled through the script file The fetching script crawls the mail data of the main page of the mailbox to obtain crawled data corresponding to the search information.

In the above process, use the crawling script in the injected script file to crawl the crawling data that matches the specified search information in the mailbox main page. For example, the search information entered by the user is credit card bill data, and the script file detects that the user After entering the information, crawl the email data in the mailbox to find the crawled data related to the credit card bill data.

It should be noted that the specified search information here can refer to the specific search information entered by the user after logging in to the main page of the mailbox, or it can refer to the saved search information that is queried regularly through background operations. The content of the search information can be modified by business personnel on the background operation page after logging in.

S130: After the crawling operation is completed, upload the crawled data to a server for analysis.

In this step, after the client completes the crawling operation, it uploads the crawled data to the server, and the server can perform in-depth text analysis based on the crawled data.

For example, in the email crawling scenario of obtaining credit card bill data, the server performs text analysis based on the crawled data to obtain credit card bill data, and returns the bill data to the main page of the mailbox, so that users can obtain the crawling results in a timely manner.

S140: Receive the parsing result returned by the server, and display the parsing result, wherein the parsing result includes target data matching the search information.

In this step, after the server completes the parsing, it receives the parsing result returned by the server, and displays the parsing result on the main page of the mailbox, and the displayed information includes target data matching the specified search information.

For example, in the email crawling scenario of obtaining credit card bill data, the server performs text analysis based on the crawled data to obtain credit card bill data, and returns the bill data to the main page of the mailbox. At this time, the content loaded on the main page of the mailbox includes getting The billing data of the received credit card, the billing data may be arranged in date order, or may be the billing data obtained after statistics are made according to different payment types.

The content of the bill data can be bill information for daily necessities, housing payment, transportation, diet, clothing and beauty, sports and health, culture, education and entertainment, communication and logistics, and other consumption.

The crawling method of the above-mentioned webpage mailbox data, when the main page of the mailbox is successfully loaded, calls the callback function of the browser, wherein the callback function includes the injected script file; obtains the specified search information, and crawls through the script file Get the script to crawl the mail data of the main page of the mailbox to obtain the crawl data corresponding to the search information; when the crawl operation is completed, upload the crawl data to the server for analysis; receive the crawl data from the server Returning the parsing result, and displaying the parsing result, wherein the parsing result includes target data matching the search information.

In this solution, the script file in the callback function is injected in advance, and after the main page of the mailbox is successfully loaded, the callback function in the browser is called. At this time, the injected script file starts to run, and the mail data in the main page of the mailbox is crawled. , in order to obtain the crawled data corresponding to the specified search information. After the crawling operation is completed, all the crawled data will be uploaded to the server for analysis. In this way, the analysis result returned by the server is the specified search information in this solution. Matching target data; this solution can directly use script files to extract user-related information on the client side, without completely crawling relevant content through the server, and avoiding the phenomenon that the server is blocked due to multiple crawling data. At the same time, The script file injected into the callback function of the browser replaces the crawling operation on the server side, which saves the resources consumed by the server side in the process of crawling data.

In one embodiment, after the main page of the mailbox is successfully loaded in step S110, the callback function of the browser is called, wherein, before the callback function includes the injected script file, it may also include:

(1) Obtain the script file from the specified directory according to the version information file of the client;

(2) Inject the script file into the callback function of the browser.

In this step, the script file is obtained from the specified directory according to the version information file of the client. It should be noted that the version information file here refers to the information file corresponding to the number of times the software program is released; the software program here can be The mailbox office software installed on the client; the specified directory here refers to the directory file under the specified path determined according to the storage path of the version information file.

As can be seen from the above description, according to the version information file of the mailbox installed on the client, the script file can be obtained from the specified directory of the version information file. The script file here can include various file types, for example: public scripts of public function libraries, Public scripts, crawling scripts, and login scripts of any door.

①The public script of the public function library: jquery-2.2.4.min.js; it can be called as the basic js function library;

②Public script of Anydoor script: native_RYM.js; it can be used for code interaction between the web page and the client’s IOS system or Android system;

The Anydoor script will determine which system the current environment is, and call the corresponding WebView interface of the system. IOS system and Android system can use the same crawling or login script;

③ Crawling script: crawl.js; it can be used to crawl webpage data. In the scenario of email crawling, this crawling script can mainly crawl email data in emails;

④Login script: autologin.js; it can be used to automatically log in when the user account and password are saved.

The above crawling scripts and login scripts need to rely on public scripts for execution, so when injecting script files, the public scripts are injected first.

After obtaining the script file, inject the script file into the callback function of the browser, further, access the mailbox login webpage through the callback function, and crawl the user login information of the mailbox login webpage through the login script of the script file.

The purpose of using the callback function to access the mailbox login webpage here is that WebView is a native control in the client system code, which can be understood as a simple browser in the system. When accessing the mailbox login webpage, it will be called when the webpage is loaded successfully. If When the user has previously saved the relevant email login account and login password, js can discover and perform crawling operations.

Preferably, the user login information is displayed, and when the confirmation information of the user login information is received from the user, the mailbox login webpage is redirected to the mailbox main page.

In this step, access the mailbox login webpage through the callback function, and crawl the user login information related to the mailbox login webpage from the mailbox login webpage through the injected script file, and then display the user login information. When confirming the information, the email login webpage will be redirected to the main email page.

In the above process, after the script file crawls the user login information saved by the user, it displays the user login information, and the user browses and confirms the display information. After receiving the confirmation information from the user, the page jumps to the main page of the mailbox noodle.

Furthermore, when the user has not saved the user login information on the email login page, there is no need to continue to crawl relevant information through the script file. After the user manually enters the account password and successfully logs in, he will jump to the main email page, and the background will also ask for the user's opinion. Save the user account and password entered by the current user.

In one embodiment, before the step of obtaining the script file from the specified directory according to the version information file of the client, it may further include:

(1) requesting a communication interface, and obtaining an updated version information file through the communication interface;

(2) comparing the updated version information file with the original version information file;

(3) When there is a comparison result, update the original version information file;

(4) When there is no comparison result, there is no need to update the original version information file.

In the above steps, the latest version information file of the script is obtained by requesting the communication interface, and compared with the locally configured version information file to determine whether an update is required.

The communication interface here can be a GP interface, which is an interface adopted between GSNs of different PLMN networks. It adds a border gateway (BG, Border Gateway) and a firewall, and provides a border gateway routing protocol through the BG to complete the belonging to different PLMNs. GPRS supports communication between nodes.

Furthermore, the version information file here can be saved as a file in json format, and the saving format is as follows: {"downloadUrl":"","jsName":"qq","md5":"279f1dd1759c2c270ea2837f1121ebfb","needUpdateVersion" :false,"updatedJsPath":"","version":"1.1.3"}.

In this embodiment, before the script file is obtained from the designated directory through the version information file of the client, the latest updated version information file is checked and obtained by requesting the GP interface, so that the updated script file can be obtained to reduce local scripts. The problem of information asymmetry caused by version aging when the file is running.

In one embodiment, the step of comparing the updated version information file with the original version information file may include:

(1) Acquiring the version number of the updated version information file and the version number of the original version information file;

(2) Compare the version number of the updated version information file with the version number of the original version information file.

In the above steps, by comparing the version numbers of different version information files, it can be known whether the locally configured version information files need to be updated.

It should be noted that different version information files have different version numbers, and the version number is used as identification information to distinguish different version information files, and is used to compare the old and new version information files, so as to further check whether the local version information files need to be updated.

In this embodiment, by comparing the version number of the acquired updated version information file with the version number of the local original version information file, so as to compare whether the pulled updated version information file is the latest version, Avoid directly pulling the updated version information file to the local, but there is no difference with the original version information file, resulting in resource consumption for no reason.

In one embodiment, after the step of determining whether the original version information file needs to be updated according to the comparison result, it may further include:

(1) When the original version information file needs to be updated, download and save the updated version information file;

(2) comparing the check value of the updated version information file with the check value of the communication interface;

(3) Determine whether the updated version information file is downloaded correctly according to the comparison result.

In this step, when it is determined that the original version information file needs to be updated through the comparison results of the above-mentioned embodiments, download and save the version information file, and then compare the check value of the version information file with the check value of the communication interface. Perform comparison to ensure that the downloaded version information file is the correct version information file.

The above-mentioned verification value can be an md5 value, and the above-mentioned communication interface can be a GP interface. When an update is required, download the latest version information file, compare the md5 value of the latest version information file with the md5 value of the GP interface, and judge whether the download is correct , when it is determined that the download is correct, store the latest version information file as configuration data.

Further, if the md5 value of the latest version information file does not match the md5 value of the GP interface, it means that the downloaded version information file is incorrect, then delete the previously downloaded and saved version information file and download it again until the download is correct So far, the terminal device 110 is prevented from being poisoned due to the Trojan horse virus contained in the downloaded incorrect version information file.

In one embodiment, the step of obtaining the script file from the specified directory according to the version information file of the client may include:

(1) searching for configuration files by specifying a directory, wherein the configuration files include a plurality of script files and configuration data corresponding to the script files;

(2) when the script file exists, obtain the configuration data corresponding to the script file, and compare the check value of the configuration data with the check value of the script file;

(3) Obtain the script file according to the comparison result.

In this embodiment, the script file is obtained from the specified directory according to the version information file of the client, which may specifically include the following steps:

Search for the configuration file in the specified data directory. If the specified directory is /data/.../app_emailcrawl/, then search for the xx.js file in the specified directory. If the script file exists, obtain the configuration data of the script file and set The md5 value of the configuration data is compared with the md5 value of the script file under /data/, and if the md5 value of the configuration data is correct, the script file under /data/ is obtained.

Further, if the md5 value of the configuration data is wrong, the script file does not exist in the specified data directory, and the script file in the specified data directory has no configuration data, delete the relevant data in the specified data directory, and copy the assets under Asset The script stores the configuration data, notifies the outside, and is handled by the outside.

In the above embodiment, the check value between the script file obtained in the specified directory and the configuration data is compared to prevent the obtained script file from being a wrong script file and avoid memory consumption.

In one embodiment, the search information includes credit card billing information;

In step S120, the step of obtaining the specified search information, and crawling the mail data of the main page of the mailbox through the crawling script of the script file, and obtaining the crawled data corresponding to the search information may include:

(1) Execute the script file according to the specified credit card bill information, wherein the script file includes a crawling script;

(2) crawling mail data related to the credit card bill information in the main page of the mailbox by using the crawling script;

(3) Count the mail data crawled by the crawling script multiple times, and obtain crawled data corresponding to the credit card bill information according to the statistical results.

In this embodiment, the specified search information is obtained, and the crawling script in the script file is run according to the search information, and the email data related to the search information is crawled in the mailbox main page by using the crawling script, and the crawled The data content in the email data is further analyzed to obtain the target data matching the search information input by the user.

Furthermore, while the script file is performing crawling, it can return the crawling status of the crawling process to the client, where the crawling status can include crawling progress, the percentage of crawling, and the time to complete the crawling Wait.

In one embodiment, as shown in FIG. 4, FIG. 4 is a schematic diagram of webpage mailbox data interaction in one embodiment. This embodiment provides a method for crawling webpage mailbox data, which may include:

The terminal device 110 uses the script file injected in the callback function of the browser. When the client opens the main page of the mailbox, after calling the callback function, the script content of the script file is executed, and the crawling script is used to crawl the main page of the mailbox. Email data, and finally crawled related crawled data are uploaded to the server 120, and the server 120 performs in-depth text analysis on the crawled data, and then returns the analysis result to the main page of the mailbox for display.

In the above process, the server 120 only needs to receive the crawling data uploaded by the terminal device 110, and does not need to visit the client multiple times, preventing the phenomenon of being blocked due to frequent visits, and further reducing the resource consumption of the server 120.

In one embodiment, as shown in FIG. 5, FIG. 5 is a schematic structural diagram of a crawling device for web mailbox data in an embodiment. A crawling device for web mailbox data is provided in this embodiment, which includes: a calling module 210, crawling module 220, data transmission module 230, information display module 240, wherein:

The calling module 210 is configured to call a callback function of the browser after the main page of the mailbox is loaded successfully, wherein the callback function includes an injected script file.

In this module, after the user logs in successfully, the page mask animation is turned on on the main mailbox main page to display the loading effect of "crawling and loading". The main purpose is to cover the main mailbox main page and prevent user operations from interfering with automatic crawling.

The purpose of using the callback function to access the email login webpage here is that the callback function is a native control in the client system code, which can be understood as a simple browser in the system. When accessing the email login webpage, it will be called when the webpage is loaded successfully. At this point, the script file has been injected into the mailbox login webpage. When the callback function is called, the js code after the script injection will be in the running state, and the user's operation will be detected in real time.

The crawling module 220 is configured to acquire specified search information, and crawl the mail data of the main page of the mailbox through the crawling script of the script file to obtain crawled data corresponding to the search information.

In this module, after the main page of the mailbox is successfully loaded by calling module 210, the callback function of the browser is called, wherein, after the callback function includes the injected script file, the specified search information is obtained, and the script file is used to obtain the specified search information. The crawling script crawls the mail data of the main page of the mailbox to obtain the crawled data corresponding to the search information.

In the above process, use the crawling script in the injected script file to crawl the crawling data that matches the specified search information in the mailbox main page. For example, the search information entered by the user is credit card bill data, and the script file detects that the user After entering the information, crawl the email data in the mailbox to find the crawled data related to the credit card bill data.

It should be noted that the specified search information here can refer to the specific search information entered by the user after logging in to the main page of the mailbox, or it can refer to the saved search information that is queried regularly through background operations. The content of the search information can be modified by business personnel on the background operation page after logging in.

The data transmission module 230 is configured to upload the crawled data to the server for analysis after the crawling operation is completed.

In this module, after the client completes the crawling operation, it uploads the crawled data to the server, and the server can perform in-depth text analysis based on the crawled data.

For example, in the email crawling scenario of obtaining credit card bill data, the server performs text analysis based on the crawled data to obtain credit card bill data, and returns the bill data to the main page of the mailbox, so that users can obtain the crawling results in a timely manner.

The information display module 240 is configured to receive the parsing result returned by the server and display the parsing result, wherein the parsing result includes target data matching the search information.

In this module, after the server finishes parsing, it receives the parsing result returned by the server, and displays the parsing result on the main page of the mailbox. The displayed information includes the target data that matches the specified search information.

For example, in the email crawling scenario of obtaining credit card bill data, the server performs text analysis based on the crawled data to obtain credit card bill data, and returns the bill data to the main page of the mailbox. At this time, the content loaded on the main page of the mailbox includes getting The billing data of the received credit card, the billing data may be arranged in date order, or may be the billing data obtained after statistics are made according to different payment types.

For example: the content of the bill data can be bill information for daily necessities, housing payment, transportation, diet, clothing and beauty, sports and health, culture, education and entertainment, communication and logistics, and other consumption.

The above crawling device for webpage mailbox data, when the main page of the mailbox is successfully loaded, calls the callback function of the browser, wherein the callback function includes the injected script file; obtains the specified search information, and crawls through the script file Get the script to crawl the mail data of the main page of the mailbox to obtain the crawl data corresponding to the search information; when the crawl operation is completed, upload the crawl data to the server for analysis; receive the crawl data from the server Returning the parsing result, and displaying the parsing result, wherein the parsing result includes target data matching the search information.

In this solution, the script file in the callback function is injected in advance, and after the main page of the mailbox is successfully loaded, the callback function in the browser is called. At this time, the injected script file starts to run, and the mail data in the main page of the mailbox is crawled. , in order to obtain the crawled data corresponding to the specified search information. After the crawling operation is completed, all the crawled data will be uploaded to the server for analysis. In this way, the analysis result returned by the server is the specified search information in this solution. Matching target data; this solution can directly use script files to extract user-related information on the client side, without completely crawling relevant content through the server, and avoiding the phenomenon that the server is blocked due to multiple crawling data. At the same time, The script file injected into the callback function of the browser replaces the crawling operation on the server side, which saves the resources consumed by the server side in the process of crawling data.

For specific limitations on the crawling device for web mailbox data, please refer to the above-mentioned limitations on the crawling method for web mailbox data, which will not be repeated here. All or part of the modules in the crawling device for webpage mailbox data can be realized by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the terminal device in the form of hardware, or can be stored in the memory of the terminal device in the form of software, so that the processor can call and execute the corresponding operations of the above modules.

In one embodiment, a terminal is proposed, including a memory 320 and a processor 380, the memory 320 stores computer-readable instructions, and when the computer-readable instructions are executed by the processor 380, the The processor 380 executes the steps in the method for crawling web mailbox data as described in any one of the above embodiments.

As shown in FIG. 6, FIG. 6 shows a partial structural block diagram of a mobile phone related to the terminal provided by the embodiment of the present invention. 6, the mobile phone includes: a radio frequency (Radio Frequency, RF) circuit 310, a memory 320, an input unit 330, a display unit 340, a sensor 350, an audio circuit 360, a wireless fidelity (wireless fidelity, WiFi) module 370, a processor 380 , and power supply 390 and other components. Those skilled in the art can understand that the structure of the mobile phone shown in FIG. 6 does not constitute a limitation to the mobile phone, and may include more or less components than shown in the figure, or combine some components, or arrange different components.

The following is a specific introduction to each component of the mobile phone in conjunction with Figure 6:

The RF circuit 310 can be used for sending and receiving information or receiving and sending signals during a call. In particular, after receiving the downlink information from the base station, it is processed by the processor 380; in addition, the designed uplink data is sent to the base station. The memory 320 can be used to store computer-readable storage instructions and modules, and the processor 380 executes various functional applications and data processing of the mobile phone by running the computer-readable storage instructions and modules stored in the memory 320 . The input unit 330 can be used to receive input numbers or character information, and generate key signal input related to user settings and function control of the mobile phone. The display unit 340 may be used to display information input by or provided to the user and various menus of the mobile phone.

The handset may also include at least one sensor 350, such as a light sensor, motion sensor, and other sensors. The audio circuit 360, the speaker 361, and the microphone 362 can provide an audio interface between the user and the mobile phone. WiFi is a short-distance wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 370, which provides users with wireless broadband Internet access. The processor 380 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing computer-readable storage instructions and/or modules stored in the memory 320, and calling the stored in the memory 320 Data, perform various functions of the mobile phone and process data, so as to monitor the mobile phone as a whole.

The mobile phone also includes a power supply 390 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 380 through the power management system, so as to realize functions such as managing charging, discharging, and power consumption management through the power management system.

In one embodiment, a storage medium is proposed. When the computer-readable storage instructions and modules stored in the memory 320 are executed by the processor 380, the processor 380 can realize the above method of crawling web mailbox data, and implement The functions of the corresponding modules in the crawling device for webpage mailbox data of the embodiment shown in FIG. 5 .

It should be understood that although the various steps in the flow chart of the accompanying drawings are displayed sequentially according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the order of execution is also It is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

The above descriptions are only part of the embodiments of the present invention. It should be pointed out that those skilled in the art can make some improvements and modifications without departing from the principles of the present invention. It should be regarded as the protection scope of the present invention.

Claims (10)

1. a kind of crawling method of webpage mailbox data, which comprises the steps of:

After mailbox homepage loads successfully, the call back function of browser is called, wherein the call back function includes the foot of injection This document；

Specified search information is obtained, and script is crawled to the mail data of the mailbox homepage by the script file It is crawled, obtains corresponding with described search information crawling data；

After the completion of crawling operation, by it is described crawl data and be uploaded to server parse；

The parsing result that the server returns is received, and the parsing result is shown, wherein the parsing result packet Include the target data to match with described search information.

2. the method according to claim 1, wherein calling browser after mailbox homepage loads successfully Before the step of call back function, further includes:

Script file is obtained from specified directory according to the version information file of client；

It will be in the call back function of script file injection browser.

3. according to the method described in claim 2, it is characterized in that, according to the version information file of client from specified directory Before the step of obtaining script file, further includes:

Communication interface is requested, updated version information file is obtained by the communication interface；

It will be compared between the updated version information file and original version information file；

Determine whether original version information file needs to update according to the comparison result.

4. according to the method described in claim 3, it is characterized in that, determining that original version is believed according to the comparison result After whether breath file needs the step of updating, further includes:

When original version information file needs to update, downloads and save the updated version information file；

It will be compared between the check value of the updated version information file and the check value of the communication interface；

Determine whether the updated version information file is downloaded correctly according to the comparison result.

5. according to the method described in claim 2, it is characterized in that, according to the version information file of client from specified directory The step of obtaining script file, comprising:

Search configuration file by specified directory, wherein the configuration file include multiple script files and with the script The corresponding configuration data of file；

In the presence of the script file, the corresponding configuration data of the script file is obtained, and by the school of the configuration data It tests between value and the check value of the script file and is compared；

Script file is obtained according to the comparison result.

6. the method according to claim 1, wherein described search information includes Credit Statement information；

Specified search information is obtained, and script is crawled to the mail data of the mailbox homepage by the script file It is crawled, obtains the step of crawling data corresponding with described search information, comprising:

The script file is executed according to specified Credit Statement information, wherein the script file includes crawling script；

Mail data relevant to the Credit Statement information is crawled in the mailbox homepage using the script that crawls；

The mail data that script repeatedly crawls is crawled described in statistics, and is obtained and the credit card account according to the statistical result Single information is corresponding to crawl data.

7. according to the method described in claim 6, it is characterized in that, receive the parsing result that the server returns, and by institute Parsing result to be stated to be shown, wherein the parsing result includes the steps that the target data to match with described search information, Include:

Obtain the parsing result that the server returns, wherein the parsing result includes and the Credit Statement information phase Matched billing data；

The billing data is shown in the mailbox homepage.

8. a kind of webpage mailbox data crawls device characterized by comprising

Calling module, for calling the call back function of browser, wherein the readjustment letter after mailbox homepage loads successfully Number includes the script file of injection；

Module is crawled, crawls script to the mailbox master for obtaining specified search information, and by the script file The mail data of the page is crawled, and obtains corresponding with described search information crawling data；

Data transmission module, for after the completion of crawl operation, by it is described crawl data and be uploaded to server parse；

Information display module, the parsing result returned for receiving the server, and the parsing result is shown, In, the parsing result includes the target data to match with described search information.

9. a kind of terminal, which is characterized in that including memory and processor, computer-readable finger is stored in the memory It enables, when the computer-readable instruction is executed by the processor, so that the processor is executed as claim 1 to 7 is any Step in the crawling method of one webpage mailbox data.

10. a kind of storage medium, it is characterised in that: be stored with computer-readable instruction, the computer in the storage medium When readable instruction is executed by one or more processors, so that one or more processors are executed as any in claim 1 to 7 The step of crawling method of the item webpage mailbox data.