CN105005600A - Preprocessing method of URL (Uniform Resource Locator) in access log - Google Patents

Abstract

The invention provides a preprocessing method of a URL (Uniform Resource Locator) in a website access log. The preprocessing method comprises the following steps of: Step 11, performing website URL collection: sorting and concluding a website URL address system; Step 12, performing URL configuration and storage: configuring the website URLs obtained through collection in the Step 11, and storing the website URLs into a URL rule storage table, wherein the URL rule storage table comprises the following fields including a URL unique code, a URL identification rule, a URL name and a URL matching sequence; Step 13, taking out information in the URL rule storage table obtained in Step 12, and performing sequencing according to the URL matching sequence in a way of ensuring a mother URL to be arranged in front of a son URL; Step 14, obtaining an access log record which includes the visitor IP, the accessing time, REFERER information and REQUEST information; Step 15, matching the REFERER and the REQUEST in each access log record in the Step 14 with the URL identification rule in the URL rule storage table obtained in the Step 13 according to the sequence obtained in the Step 13; and Step 16, obtaining records of the REFERER and the REQUEST which are not successfully matched in the Step 15 and are coded into -1 or null values.

Description

A Preprocessing Method of URL in Access Log

technical field

The invention relates to the field of website analysis, in particular to a method and device for preprocessing website access log URLs.

Background technique

Website access path analysis provides important data support and guidance for optimizing website structure and page layout, and understanding visitor behavior preferences. The basic data of website path analysis comes from the website's access log, which records information such as visitor's IP, access time, REFERER (the page visited last time), REQUEST (the page currently visited) and so on. Among them, REFERER and REQUEST are very important information for constructing a set of visited webpages and an access path.

The REFERER and REQUEST recorded in the access log are all in the form of URL addresses. For example, the URL (uniform resource locator, that is, the address of the WWW page) of the home page of Made in China (hereinafter referred to as: MIC) is

"www.made-in-china.com". A problem will be encountered when performing path analysis based on the original REFERER and REQUEST recorded in the access log. The REFERER and REQUEST are too detailed, which is not conducive to subsequent statistical analysis and extraction of access paths. For example, visitors to MIC mainly enter the MIC search list page through GOOGLE, and the URL addresses of the search list pages corresponding to different search words or search conditions are different. For example, if you use "led" to search, the URL of the search list page is

"www.made-in-china.com/productdirectory.do?word=led&subaction=hunt&style=b&mode=and&code=0&comProvince=nolimit&order=0&isOpenCorrection=1".

Use "led light" to search, the URL of the search list page is

"www.made-in-china.com/productdirectory.do?subaction=hunt&style=b&mode=and&code=0&comProvince=nolimit&order=0&isOpenCorrection=1&word=led+light", in fact, when we analyze the access path, it will be similar to The above two specific URLs are summarized and classified to a certain extent, for example, they are both identified as "MIC search list pages", so that the visitor access path of the entire website can be analyzed.

At present, the research on the access path mainly focuses on how to collect the set of pages visited by each visitor and construct the access path. There is little mention of how to preprocess REFERER and REQUEST in the access log, and this step is an important prerequisite for constructing the access path.

HTTP Referer is a part of the header. When the browser sends a request to the web server, it will usually bring the Referer to tell the server the address from which page the current page is redirected from (which page is linked from, that is, the address of the last visit) page), the server can obtain some information for processing. REQUESTheader is a client (usually a browser) that sends a command line (accessed page URL) when sending a request to a web server.

Contents of the invention

Purpose of the invention: the present invention provides a method and device for preprocessing the original URL recorded in the website access log, so as to solve the data preprocessing problem of REFERER and REQUEST recorded in the access log in path analysis.

A method for preprocessing website access log URLs, the steps comprising:

S11: Website URL collection, that is, sorting and summarizing the website URL address system; collecting the main or important page URLs of the website, and confirming the basic information of these URLs, including URL identification rules and URL names; among them, URL identification rules refer to According to the URL analysis and induction of the original webpage, the composition characteristics of the URL of a certain type of page are obtained; the URL identification rules can be described using regular expressions;

S12: URL configuration and storage, configuring and storing the URL of the website collected in S11 into the URL rule storage table; the URL rule storage table includes the following fields: URL unique code, URL identification rule, URL name, URL matching order; wherein, "URL unique code" is used to mark the unique identity of each URL identification rule, which is automatically generated by the database; "URL identification rule" and "URL name" are derived from step S11; "URL matching order" is used to control the URL matching order;

The determination method of "URL matching order" is: assuming that B_URL is a substring of A_URL, it is said that there is a string containment relationship between A_URL and B_URL, where A_URL is the parent URL and B_URL is the sub-URL; then the matching of A_URL and B_URL The order is A_URL first, B_URL last, that is, the parent URL is ranked before the child URL;

S13: Take out the information in the URL rule storage table obtained in S12, and sort according to the "URL matching order", so as to ensure that the parent URL is ranked before the child URL;

S14: obtain the record of visit log, comprise visitor's IP, visit time, REFERER (page visited last time), REQUEST (page visited) information;

S15: Match the REFERER and REQUEST in each access log record in S14 with the URL identification rules in the URL rule storage table acquired in S13 according to the order of acquisition in S13; if the matching is successful, record the corresponding URL identification rules The unique encoding of the URL is used as the encoding of REFERER and the encoding of REQUEST; if REFERER or REQUEST cannot match any of the URL identification rules, -1 or empty value is used as the encoding of REFERER or REQUEST;

Preferably, in step S15, the present invention matches REFERER and REQUEST in each access log record in step S14 with the URL identification rules in the URL rule storage table acquired in S13 according to the sorting in S13, including:

When REFERER or REQUEST and the URL identification rule in the URL rule storage table have a string inclusion relationship, that is, REFERER or REQUEST is the parent URL of the URL identification rule, it means that REFERER or REQUEST matches the URL identification rule successfully;

If REFERER or REQUEST can be successfully matched with multiple URL identification rules in the URL rule storage table, then take the URL identification rule ranked first according to the sorting in S13;

S16: Obtain the unmatched REFERER and REQUEST in S15, that is, records whose REFERER code or REQUEST code is -1 or null, and merge all unmatched REFERERs and REQUESTs together to obtain an unmatched URL set;

S17: Statistically analyze the unmatched URL set in S16, obtain the URL with the largest number of unmatched URL sets, (and in combination with manual judgment and monitoring,) match URLs that do not have a successful match into the URL rule configuration table, thereby It can continuously improve the URL identification rules in the URL rule configuration table.

The present invention provides a URL preprocessing device in an access log, and its features include:

URL collection unit: used to collect URLs of websites, and to determine URL identification rules and URL names; wherein, URL identification rules refer to the compositional characteristics of URLs of a certain type of page obtained by analyzing and summarizing URLs of original web pages, and URL identification rules Rules can be described using regular expressions;

URL rule storage unit: used to store URL identification rules and related information, including: URL unique code, URL identification rules, URL name, and URL matching sequence. Among them, "URL unique code" is used to mark the unique identity of each URL identification rule; "URL identification rule" and "URL name" are derived from the URL collection unit; they are used to control the URL matching sequence.

Preferably, the URL rule storage unit includes:

URL rule configuration module: used to determine "URL unique encoding" and "URL matching sequence". The "unique URL code" can be automatically generated by the database or manually, as long as there is a one-to-one relationship between the unique URL code and the URL identification rule. The determination method of "URL matching order" is: assuming that B_URL is a substring of A_URL (for example, A_URL is "abcd" and B_URL is "abc"), it is said that there is a string containment relationship between A_URL and B_URL, where A_URL is Parent URL, B_URL is child URL. The matching order of A_URL and B_URL is that A_URL comes first and B_URL comes after, that is, the parent URL comes before the child URL.

URL rule storage module: used to store URL rule storage table, including URL identification rules and related information, including: URL unique encoding, URL identification rules, URL name, URL matching sequence;

URL rule acquisition unit: sort the URL identification rules according to the URL matching order, and acquire URL identification rules and URL unique codes in this order;

Log record acquisition unit: used to obtain each record in the access log, including visitor's IP, access time, REFERER (the page visited last time), REQUEST (the page visited) and other information;

URL matching unit: used to match the REFERER and REQUEST of each record in the access log with URL identification rules. Take out a log record, and match REFERER or REQUEST with the URL recognition rules one by one in the order in which the URL recognition rules were obtained. If REFERER or REQUEST is the parent URL of a certain URL recognition rule, the match is successful and the URL of the URL recognition rule is taken out The unique code is used as REFERER code or REQUEST code; if REFERER or REQUEST does not have a string inclusion relationship with any URL recognition rule, then special mark the REFERER code or REQUEST code, such as marking "-1" or a null value, so far Complete the matching of this log record and jump out of this matching. Then, take the next log record and perform matching according to the above method until all log records are matched;

Matching result set storage unit: used to store the matching results of the access log and URL identification rules, including: the original information in the access log such as IP, access time, REFERER, REQUEST, etc., as well as the above-mentioned REFERER code and REQUEST code;

The unmatched URL monitoring unit includes: unmatched data acquisition unit: used to obtain unmatched REFERER and REQUEST in the matching result set, and merge them into an unmatched URL set; unmatched data statistics module: count each unmatched URL set The number of URL records; unmatched data monitoring module: According to the number of records of each URL in the unmatched URL set and sorted in descending order according to the number of records, the unmatched URL set can be collected. Combined with actual business needs, it can be determined whether to configure these URLs into the URL rule storage table. If configuration is required, return to the URL collection unit and follow the above process until all URLs that need to be analyzed are added to the URL rule storage table. middle.

Beneficial results of the present invention are as follows: the present invention provides a kind of preprocessing method of the original URL recorded in the website access log, which can solve the data preprocessing problem of REFERER and REQUEST recorded in the access log in path analysis:

1) By collecting the page URL of the website and forming a website URL rule storage table, matching the REFERER and REQUEST recorded in the original access log with the URL identification rules in the URL rule storage table, encoding and naming each REFERER and REQUEST, Transform the original URL address formats of REFERER and REQUEST into codes and business names that are convenient for subsequent statistical analysis and application.

2) Through the monitoring and analysis of the unmatched URL set, the URL rule storage table can be continuously improved, and the URL rule storage table can gradually and comprehensively cover all website pages, so as to ensure that the records in the access log match as much as possible. REFERER encoding and REQUEST encoding. Provide complete preprocessed data for subsequent analysis based on access logs.

Description of drawings

Fig. 1 is a flow chart of a method for preprocessing a website access log URL according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of an apparatus for preprocessing website access log URLs according to an embodiment of the present invention.

specific implementation plan

The specific embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings and examples. Obviously, the described examples are only some of the examples of the present invention, not all of them. Changes or equivalent changes made based on the embodiments of the present application and the technical essence of the claims of the present invention still fall within the protection scope of the present application.

Referring to shown in Figure 1, the implementation steps of the present application are as follows:

S11: Website URL collection, that is, sorting and summarizing the website URL address system. The collection of website URLs can rely on manual collection at the initial stage, by manually collecting the main or important page URLs of the website, and confirming the basic information of these URLs, including URL identification rules, URL names, etc. Wherein, the URL identification rule refers to the constituent features of the URL of a certain type of webpage obtained by analyzing and summarizing the URL of the original webpage.

For example, the URL address of the product search list page of Made-in-China.com all starts with "www.made-in-china.com/productdirectory.do?"; the identification rule for the product search list page is "www.made-in-china.com/productdirectory.do?" china.com/productdirectory.do?”. Moreover, URL identification rules can be described using regular expressions.

The URL address of the product search list page is as follows:

www.made-in-china.com/productdirectory.do? word=led&subaction=hunt&style=b&mode=and&code=0&comProvince=nolimit&order=0&isOpenCorrection=1, characterized by

It starts with "www.made-in-china.com/productdirectory.do?", and the following parameters such as "word" record the search terms used and other information. Then it can be based on that if a URL starts with "www.made-in-china.com/productdirectory.do?", then the URL is a product search list page.

The URL address of the home page of Made-in-China.com is: www.made-in-china.com.

The URL address of the special event home page of Made-in-China.com is: www.made-in-china.com/special.

The URL address of the special event detail page of Made in China.com is: (such as the magic-show topic)

www.made-in-china.com/special/magic-show/.

Then the URL identification rules and URL names corresponding to the above four pages can be:

"www.made-in-china.com/productdirectory.do?", "product search list page";

"www.made-in-china.com$", "MIC homepage";

"www.made-in-china.com/special", "special homepage";

"www.made-in-china.com/special/", "special detail page".

Among them, the "$" in the recognition rules on the MIC homepage is a regular expression expression, which means that it ends with the character string before "$", here it means all characters ending with "www.made-in-china.com" string;

S12: URL configuration and storage, configuring and storing the URL of the website collected in S11 into a URL rule storage table. The URL rule storage table includes the following fields: URL unique code, URL identification rule, URL name, and URL matching sequence. Among them, "URL unique code" is used to mark the unique identity of each URL identification rule, which can be automatically generated by the database; "URL identification rule" and "URL name" come from step S11; "URL matching order" is used to control the URL matching order .

The determination method of "URL matching order" is: assuming that B_URL is a substring of A_URL (for example, A_URL is "abcd" and B_URL is "abc"), it is said that there is a string containment relationship between A_URL and B_URL, where A_URL is Parent URL, B_URL is child URL. The matching order of A_URL and B_URL is that A_URL comes first and B_URL comes after, that is, the parent URL comes before the child URL.

Specifically, if the URL of the first page of the configuration website is stored in the URL rule storage table, taking the product search list page of Made in China as an example, the URL unique code, URL identification rule, URL name, and URL matching sequence are:

"1001", "www.made-in-china.com/productdirectory.do?", "Product Search List Page", "Product Search List Page". It should be noted that currently there is no configured URL identification rule in the URL rule storage table, so the value of "URL matching order" can be arbitrary, and here the URL name can be used as the value of URL matching order.

The URL unique codes, URL identification rules, URL names, and URL matching sequences of the above four pages are:

"1002", "www.made-in-china.com/productdirectory.do?", "product search list page", "product search list page";

"1003", "www.made-in-china.com$", "MIC homepage", "MIC homepage";

"1004", "www.made-in-china.com/special", "special home page", "special page 2";

"1005", "www.made-in-china.com/special/", "Special detail page", "Special page 1".

Among them, the URL identification rule of the special detail page (www.made-in-china.com/special/) is the parent URL of the identification rule of the special home page (www.made-in-china.com/special), so the special detail page The matching order of URLs with the homepage of the topic is "thematic page 1" and "thematic page 2", which ensures that when sorting in ascending order according to the URL matching order, the topic detail page is ranked before the topic homepage.

S13: Take out the information in the URL rule storage table obtained in S12, and sort according to the "URL matching order", so as to ensure that the parent URL is ranked before the child URL.

Specifically, take out the information in the above URL rule storage table, and arrange them in ascending order according to the "URL matching order", to obtain:

"1003", "www.made-in-china.com$", "MIC homepage", "MIC homepage";

"1002", "www.made-in-china.com/productdirectory.do?", "product search list page", "product search list page";

"1005", "www.made-in-china.com/special/", "Special detail page", "Special page 1";

"1004", "www.made-in-china.com/special", "Special Home Page", "Special Page 2".

S14: Obtain the records of the access log, including information such as the visitor's IP, access time, REFERER (page visited last time), REQUEST (page visited). Specifically, the records in the access log can be in the following form:

192.168.1.1, 2015-01-01 12:01:00, www.made-in-china.com, www.google.com;

192.168.1.1, 2015-01-01 12:01:30, www.made-in-china.com/special/vacuum-pump/, www.made-in-china.com;

192.168.1.1, 2015-01-01 12:01:30, sourcing.made-in-china.com/suppliers.html, www.made-in-china.com/special/vacuum-pump/;

192.168.2.1, 2015-01-01 12:02:10, www.made-in-china.com, www.google.com;

192.168.2.1, 2015-01-01 12:03:10,

http://www.made-in-china.com/productdirectory.do? word=led&subaction=hunt&style=b&mode=and&code=0&comProvince=nolimit&order=0&isOpenCorrection=1,

www.made-in-china.com;

Among them, "192.168.1.1" and "192.168.2.1" are the IP addresses of visitors; the time close to the IP address is the visit time of the visitor to the corresponding page; the URL address adjacent to the visit time is the URL of the page currently visited by the visitor That is, REQUEST, such as www.made-in-china.com in the first record; the URL address after the URL of the currently visited page is the URL of the previous page visited by the visitor, that is, REFERER, such as www in the first record .google.com. That is to say, the visitor jumps from the last visited page (REFERER) to the current visited page (REQUEST), that is, the visitor jumps from www.google.com to www.made-in-china.com.

S15: Match the REFERER and REQUEST in each access log record in S14 with the URL identification rules in the URL rule storage table acquired in S13 according to the order acquired in S13. If the match is successful, record the unique code of the URL corresponding to the URL identification rule as the code of REFERER and the code of REQUEST. If REFERER or REQUEST cannot match any of the URL identification rules, take -1 or null value as the encoding of REFERER or REQUEST.

Preferably, in this application, in S15, the REFERER and REQUEST in each access log record in S14 are respectively matched with the URL identification rules in the URL rule storage table obtained in S13 according to the order in S13. include:

When REFERER or REQUEST and the URL identification rule in the URL rule storage table have a character string inclusion relationship, that is, REFERER or REQUEST is the parent URL of the URL identification rule, it means that REFERER or REQUEST matches the URL identification rule successfully.

If the REFERER or REQUEST can be successfully matched with multiple URL identification rules in the URL rule storage table, the URL identification rule ranked first according to the sorting in S13 is taken.

Specifically, the log records listed in S14 are matched with the URL rule storage table in S13.

Get the first record:

192.168.1.1, 2015-01-01 12:01:00, www.made-in-china.com, www.google.com;

REQUES is www.made-in-china.com, which can match the MIC home page in the URL rule storage table of S13, and take the unique URL code "1003" corresponding to the MIC home page as the REQUEST code of this record. REFERFER is www.google.com, which cannot match any URL identification rule in the URL rule storage table of S13, so set "-1" as the REFERER code of this record.

Take out the second record:

192.168.1.1, 2015-01-01 12:01:30, www.made-in-china.com/special/vacuum-pump/, www.made-in-china.com;

REQUES is www.made-in-china.com/special/vacuum-pump/, which can match the topic detail page and topic homepage in the URL rule storage table of S13 at the same time, and take the identification rule ranked first in the matching order, That is, take the unique URL code "1005" corresponding to the topic detail page as the REQUEST code of this record. The REFERFER is www.made-in-china.com, which successfully matches the MIC home page in the URL rule storage table of S13, and the REFERER code of this record is "1003".

Follow this method until all log records are matched. Finally, the matching results of all records are as follows (IP, access time, REQUEST, REFERER, REQUEST encoding, REFERER encoding):

192.168.1.1, 2015-01-01 12:01:00, www.made-in-china.com, www.google.com, 1003, -1;

192.168.1.1, 2015-01-01 12:01:30, www.made-in-china.com/special/vacuum-pump/, www.made-in-china.com, 1005, 1003;

192.168.1.1, 2015-01-01 12:01:30, sourcing.made-in-china.com/suppliers.html, www.made-in-china.com/special/vacuum-pump/, -1, 1005;

192.168.2.1, 2015-01-01 12:02:10, www.made-in-china.com, www.google.com, 1003, -1;

192.168.2.1, 2015-01-01 12:02:10,

http://www.made-in-china.com/productdirectory.do? word=led&subaction=hunt&style=b&mode=and&code=0&comProvince=nolimit&order=0&isOpenCorrection=1,

www.made-in-china.com, 1002, 1003;

S16: Obtain unmatched REFERERs and REQUESTs in S15, that is, records whose REFERER code or REQUEST code is -1 or null, and combine all unmatched REFERERs and REQUESTs to obtain an unmatched URL set.

Specifically, the REFERERs that did not match successfully in S15 are:

www.google.com;

www.google.com.

The REQUEST that did not match successfully is:

sourcing.made-in-china.com/suppliers.html.

Combine to get the set of unmatched URLs:

www.google.com;

www.google.com;

sourcing.made-in-china.com/suppliers.html.

S17: Statistically analyze the unmatched URL sets in S16 to obtain the URL with the largest number of unmatched URL sets, and combine manual judgment and monitoring to match unmatched URLs to the URL rule configuration table, so as to continuously Improve the URL identification rules in the URL rule configuration table.

Specifically, the unmatched URLs in the unmatched URL set of S16 are: www.google.com, sourcing.made-in-china.com/suppliers.html. Among them, www.google.com is the main search engine and the access portal for most foreign visitors. It should be the source of visits that most websites should focus on. Therefore, www.google.com can also be collected and configured in the URL rule storage table, then S11 to S13 are repeated.

The present invention also provides a URL preprocessing device in the access log, which is equipped with various functional modules in the above method, URL collection unit: used to collect URLs of websites, and determine URL identification rules, URL names, etc. Among them, the URL identification rule refers to the composition characteristics of the URL of a certain type of page based on the analysis and induction of the URL of the original web page. For example, the URL address of the product search list page of Made in China. china.com/productdirectory.do?”, then the identification rule for the product search list page is “www.made-in-china.com/productdirectory.do?”. Moreover, URL identification rules can be described using regular expressions.

URL rule storage unit: used to store URL identification rules and related information, including: URL unique code, URL identification rules, URL name, and URL matching sequence. Among them, "URL unique code" is used to mark the unique identity of each URL identification rule; "URL identification rule" and "URL name" are derived from the URL collection unit; they are used to control the URL matching sequence.

Log record obtaining unit: used to obtain each record in the access log, including visitor's IP, access time, REFERER (the page visited last time), REQUEST (the page visited) and other information.

URL matching unit: used to match the REFERER and REQUEST of each record in the access log with URL identification rules. Take out a log record, and match REFERER or REQUEST with the URL recognition rules one by one in the order in which the URL recognition rules were obtained. If REFERER or REQUEST is the parent URL of a certain URL recognition rule, the match is successful and the URL of the URL recognition rule is taken out The unique code is used as REFERER code or REQUEST code; if REFERER or REQUEST does not have a string inclusion relationship with any URL recognition rule, then special mark the REFERER code or REQUEST code, such as marking "-1" or a null value, so far Complete the matching of this log record and jump out of this matching. Then, take the next log record and perform matching according to the above method until all log records are matched.

Matching result set storage unit: used to store the matching results of access logs and URL identification rules, including: original information in access logs such as IP, access time, REFERER, REQUEST, etc., as well as the above-mentioned REFERER codes and REQUEST codes.

Unmatched URL monitoring unit: It is used to analyze the unmatched REFERER and REQUEST in the access log, so as to improve the URL identification rules to cover all pages of the website, and achieve the purpose of gradual improvement and optimization.

Unmatched URL monitoring units include:

Unmatched data acquisition unit: used to acquire unmatched REFERER and REQUEST in the matching result set, and merge them into an unmatched URL set.

Unmatched data statistics module: count the number of records of each URL in the unmatched URL set.

Unmatched data monitoring module: According to the record number of each URL in the unmatched URL set and sort them in descending order according to the number of records, the unmatched URL set can be collected. Combined with actual business needs, it can be determined whether to configure these URLs into the URL rule storage table. If configuration is required, return to the URL collection unit and follow the above process until all URLs that need to be analyzed are added to the URL rule storage table. middle.

The method and system provided by the present invention have been described in detail above, but these descriptions cannot be interpreted as limiting the scope of the present invention. The protection scope of the present invention is defined by the appended claims. All changes above are within the protection scope of the present invention.

Claims (3)

1. A preprocessing method of website access log URL is characterized in that it comprises steps: S11: Website URL collection, that is, sorting and summarizing the website URL address system; collecting the main or important page URLs of the website, and confirming the basic information of these URLs, including URL identification rules and URL names; among them, URL identification rules refer to According to the URL analysis and induction of the original webpage, the composition characteristics of the URL of a certain type of page are obtained; the URL identification rules can be described using regular expressions; S12: URL configuration and storage, configuring and storing the URL of the website collected in S11 into the URL rule storage table; the URL rule storage table includes the following fields: URL unique code, URL identification rule, URL name, URL matching order; wherein, "URL unique code" is used to mark the unique identity of each URL identification rule, which is automatically generated by the database; "URL identification rule" and "URL name" are derived from step S11; "URL matching order" is used to control the URL matching order; The determination method of "URL matching order" is: assuming that B_URL is a substring of A_URL, it is said that there is a string containment relationship between A_URL and B_URL, where A_URL is the parent URL and B_URL is the sub-URL; then the matching of A_URL and B_URL The order is A_URL first, B_URL last, that is, the parent URL is ranked before the child URL; S13: Take out the information in the URL rule storage table obtained in S12, and sort according to the "URL matching order", so as to ensure that the parent URL is ranked before the child URL; S14: obtain the record of visit log, comprise visitor's IP, visit time, REFERER (page visited last time), REQUEST (page visited) information; S15: Match the REFERER and REQUEST in each access log record in S14 with the URL identification rules in the URL rule storage table acquired in S13 according to the order of acquisition in S13; if the matching is successful, record the corresponding URL identification rules The unique encoding of the URL is used as the encoding of REFERER and the encoding of REQUEST; if REFERER or REQUEST cannot match any of the URL identification rules, -1 or empty value is used as the encoding of REFERER or REQUEST; S16: Obtain the unmatched REFERER and REQUEST in S15, that is, records whose REFERER code or REQUEST code is -1 or null, and merge all unmatched REFERERs and REQUESTs together to obtain an unmatched URL set; S17: Perform statistical analysis on the unmatched URL set in S16, obtain the URL with the largest number of unmatched URL sets, and match the URLs that have not been successfully matched into the URL rule configuration table, so as to continuously improve the URL identification in the URL rule configuration table rule. 2. the preprocessing method of website access log URL according to claim 1 is characterized in that step S15 is with the URL in the URL rule storage table that obtains in S13 with REFERER and REQUEST in each access log record in step S14 respectively The identification rules are matched according to the order in S13, including: When REFERER or REQUEST and the URL identification rule in the URL rule storage table have a string inclusion relationship, that is, REFERER or REQUEST is the parent URL of the URL identification rule, it means that REFERER or REQUEST matches the URL identification rule successfully; If the REFERER or REQUEST can be successfully matched with multiple URL identification rules in the URL rule storage table, the URL identification rule ranked first according to the sorting in S13 is taken. 3. A URL preprocessing device in an access log, characterized in that it comprises: URL collection unit: used to collect URLs of websites, and to determine URL identification rules and URL names; wherein, URL identification rules refer to the compositional characteristics of URLs of a certain type of page obtained by analyzing and summarizing URLs of original web pages, and URL identification rules Rules can be described using regular expressions; URL rule storage unit: used to store URL identification rules and related information, including: URL unique encoding, URL identification rules, URL name, URL matching sequence; where, "URL unique encoding" is used to mark the uniqueness of each URL identification rule Identity; "URL identification rule" and "URL name" are derived from the URL collection unit; used to control the URL matching order; Log record acquisition unit: used to obtain each record in the access log, including visitor's IP, access time, REFERER, REQUEST information; URL matching unit: used to match the REFERER and REQUEST of each record in the access log with URL identification rules. Take out a log record, and match REFERER or REQUEST with the URL recognition rules one by one in the order in which the URL recognition rules were obtained. If REFERER or REQUEST is the parent URL of a certain URL recognition rule, the match is successful and the URL of the URL recognition rule is taken out The unique code is used as REFERER code or REQUEST code; if REFERER or REQUEST does not have a string inclusion relationship with any URL recognition rule, then special mark the REFERER code or REQUEST code, such as marking "-1" or a null value, so far Complete the matching of this log record and jump out of this matching. Then, take the next log record and perform matching according to the above method until all log records are matched; Matching result set storage unit: used to store the matching results of the access log and URL identification rules, including: the original information in the access log such as IP, access time, REFERER, REQUEST, etc., as well as the above-mentioned REFERER code and REQUEST code; The unmatched URL monitoring unit includes: unmatched data acquisition unit: used to obtain unmatched REFERER and REQUEST in the matching result set, and merge them into an unmatched URL set; unmatched data statistics module: count each unmatched URL set The number of URL records; unmatched data monitoring module: According to the number of records of each URL in the unmatched URL set and sorted in descending order according to the number of records, the unmatched URL set can be collected. Combined with actual business needs, it can be determined whether to configure these URLs into the URL rule storage table. If configuration is required, return to the URL collection unit and follow the above process until all URLs that need to be analyzed are added to the URL rule storage table. middle. URL rule storage units include: URL rule configuration module: used to determine "URL unique code" and "URL matching order"; where "URL unique code" is automatically generated by the database, or manually generated, as long as the URL unique code and URL identification rules are in a one-to-one relationship ; The determination method of "URL matching order" is as follows: assuming that B_URL is a substring of A_URL, it is said that there is a string containment relationship between A_URL and B_URL, wherein A_URL is the parent URL and B_URL is the sub-URL. The matching order of A_URL and B_URL is that A_URL comes first and B_URL comes after, that is, the parent URL comes before the child URL; URL rule storage module: used to store URL rule storage table, including URL identification rules and related information, including: URL unique encoding, URL identification rules, URL name, URL matching sequence; URL rule acquisition unit: sort the URL identification rules according to the URL matching order, and acquire URL identification rules and URL unique codes in this order.