CN106339459B - The method that Chinese web page is presorted is carried out based on Keywords matching - Google Patents

Abstract

The invention relates to a method for pre-classifying Chinese webpages based on keyword matching. In the process of making the training set required by the classification algorithm, the method manually marks each training webpage, and at the same time, uses keywords representing the webpage in the webpage. Also mark it out to generate a keyword table; for each test webpage, first extract the keywords that appear in the webpage according to the keyword table, and then transfer the labels of the training set to the test by performing keyword matching calculations with the training set. webpage; if the pre-classification method fails to give the classification result of the training webpage, the test webpage needs to be further classified. This method reduces the running time of computationally complex classification techniques such as SVM, KNN, and Naive Bayesian, and also improves the accuracy and recall of classification results.

Description

A Method of Pre-classifying Chinese Web Pages Based on Keyword Matching

technical field

The invention relates to information processing in the computer field, in particular to a method for pre-classifying Chinese webpages based on keyword matching.

Background technique

With the rapid development of the Internet, the information stored in the form of webpages is still growing explosively, so classifying webpage information has become one of the indispensable methods for people to obtain useful information. The current mainstream classification algorithms include SVM, KNN, and Naive Bayesian algorithms. Among them, SVM requires very little training set, and its classification effect on English web pages is also very good. The accuracy rate and recall rate of the classification results of the Chinese webpage classification system based on SVM technology cannot meet the requirements. This is because English has natural delimiters, while Chinese can only be segmented using the Chinese tokenizer before quantifying the webpage text. However, no matter how good the Chinese word segmentation device is, it cannot make the word segmentation completely accurate, which greatly affects the effect of Chinese web page classification.

Contents of the invention

In view of the above problems, the present invention proposes a method for pre-classifying Chinese webpages based on keyword matching, which can greatly reduce the running time of mainstream classification techniques, and at the same time improve the accuracy and recall of classification results.

The technical scheme that the present invention solves the problems of the technologies described above is as follows:

A method for pre-classifying Chinese webpages based on keyword matching, comprising the following steps:

1) mark the category label TAG of each training webpage TR in the training set TRS and the keyword set KWS representing the category of the webpage, and generate the keyword table KWT;

2) extract the keywords contained in each test webpage TE in the test set TES according to the KWT, and form the keyword set TEK;

3) Calculate the TEC of each two-tuple (keyword pair) of the TEK and traverse the training set, transfer the TAG of the TR containing the TEC in the KWS to the TE corresponding to the TEC, and store the TAG in the TE's In the label set TAGS;

4) Perform frequency statistics on the tags in TAGS, and take the tags with the highest frequency according to the requirements as the pre-classified tags of the test web page.

On the basis of the above technical solutions, the present invention can also be improved as follows.

Further, step 1) also includes generating a keyword table KWT after deduplicating the keyword sets KWS of all training webpages belonging to the same category.

Further, the above-mentioned concrete steps of generating the keyword table KWT are as follows:

1-1) Create a new mapping M, use each keyword K of the KWS of the first training webpage TR as the key of M, and set the corresponding initial value to 1;

1-2) For each keyword K in the KWS of the second TR, first judge whether K is already included in M, if it exists, add 1 to the value of the key-value pair whose key is K; if not, then Add the key-value pair <K,1> to M;

1‐3) For the remaining TRs, repeat steps 1-2) until the last TR;

1-4) Set a threshold s, when the value of the key-value pair is less than s, set the value to 0; otherwise, set the value of the key-value pair to 1.

Further, the specific steps of calculating the binary group TEC of TEK in step 2) are as follows:

3-1-1) For a TEK containing N keywords K1, K2, ..., KN, search for keyword pairs in the following order, and each keyword pair needs to enter step 3-1-2) for judgment : Keyword pairs containing K1 are <K1,K2>,<K1,K3>,...,<K1,KN>, keyword pairs containing K2 are <K2,K3>,...,<K2,KN> , until the keyword pair <K(N-1), KN> containing K(N-1) is reached;

3-1-2) If at least one of the current keyword pairs satisfies its corresponding value in M to be 1, traverse the training set TRS for TEC; if not, go back to step 3-1-1) and find The next keyword pair.

Further, in step 2), at least one keyword in TEC of all binary groups (namely keyword pairs) in TEK is an important keyword, and the important keyword is the keyword with the highest frequency of occurrence.

Further, step 2) also includes carrying out frequency statistics to the keywords that appear in the test webpage, when the frequency of occurrence of the secondary keywords exceeds the set threshold, it can be marked as an important keyword, and when calculating the test webpage When keyword binary groups are used, more binary groups are obtained, and the accuracy and recall rate of classification results are improved.

Among them, "secondary keywords" are keywords that did not appear very frequently in the first statistics, but with the increase of statistics, the frequency of occurrence may change and become keywords with high frequency of occurrence. "Important keywords" are the keywords with the highest frequency of occurrence, and the level is 0. The classification of secondary and important is manually marked, but a threshold is also set to distinguish them. For example, 0.2 is the threshold value. When the keyword marked as secondary, it will become an important keyword as the statistical frequency increases and exceeds 0.2. Initially, the method of human labeling is adopted.

Further, the process of each TEC traversing the training set TRS in step 3) specifically includes:

3-2-1) If the KWS of TR contains the first keyword of TEC, go to step 3-2-2); otherwise, calculate the next TEC of TEK and start traversing the training set again;

3-2-2) If the KWS of the TR contains at least one important keyword of the TEC, add the TAG of the TR to the tag set TAGS of the TE. If the TAG is already included in TAGS, add 1 to the value corresponding to the key-value pair with the TAG as the key; otherwise, add the <TAG,1> key-value pair to TAGS.

Further, step 4) also includes performing classification calculation on test web pages TE that do not contain keywords (that is, test web pages that fail pre-classification).

The beneficial effects of the present invention are:

1. While manually labeling the category labels of the training set, give the keywords that characterize the category, then perform frequency statistics on all keywords, set thresholds, divide keywords into important and secondary, and get key words glossary. Dividing keywords into important and secondary levels according to frequency of occurrence can make full use of the frequency information of keywords, make the keyword binary groups of the test webpage better reflect the attributes of the webpage itself, and improve the accuracy of Chinese webpage classification.

2. For each test webpage in the test set, first traverse the keyword table to obtain the keyword set contained in the webpage; then find all the binary groups of the keyword set, and at least one keyword in the binary group is required to be important The candidate label of the test webpage obtained by matching the keyword pair is more accurate, which also improves the accuracy of the webpage classification result.

3. Afterwards, for each binary group, traverse the training set, if the keyword set of the training webpage contains the binary group, then add the label of the training webpage to the label set of the test webpage; finally, the label of the test webpage The concentrated tags are used for frequency statistics, and the tags with the highest frequency are selected as the pre-classified tags of the test web page according to the needs. Giving the training web pages multiple labels reasonably improves the accuracy and recall of the classification results.

4. Due to the small number of training sets, the time consumed by the whole process increases linearly with the size of the test set. This greatly reduces the running time of Chinese webpage classification, and at the same time improves the accuracy and recall of classification results.

Description of drawings

Figure 1 is a structural diagram of the training set and the test set.

FIG. 2 is a flowchart of a method for pre-classifying Chinese web pages based on keyword matching.

Detailed ways

The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

A Chinese web page classification system is implemented based on SVM technology. The provided training set and test set are TRS and TES respectively, as shown in Figure 1, where TRS is decomposed into several TRs, and TRID refers to the number of each TR; TES is decomposed into several TEs, and TEID refers to the number of each TR. TE number, TECS refers to the set of two-tuple TEC; KET is the keyword table, including the keyword KW and the value of the corresponding key-value pair.

The method of pre-classifying Chinese web pages based on keyword matching, that is, the steps of pre-classifying TES through TRS is shown in Figure 2, and the details are as follows:

Step 1: For each training webpage TR in the training set TRS, mark its category label TAG and the keyword set KWS representing the category of the webpage, repeat step 1 until all training sets are marked;

Step 2: Deduplicate the KWS of all training webpages, store them in the keyword table KWT, and proceed to step 3;

Step 3: For each test web page TE in the test set TES, traverse the KWT, search for the keywords contained in the TE, and form the keyword set TEK. If TEK is not empty, go to step 4; if TEK is empty, go to step 8;

Step 4: Calculate the first two-tuple TEC of TEK, that is, the keyword pair, and go to step 5;

Step 5: For TEC, traverse the training set TRS, if the KWS of TR includes TEC, transfer the TAG of this TR to the TE, store it in the tag set TAGS of the TE, and count the number of occurrences of the tag. Go to step 6;

Step 6: Repeat steps 4 and 5 until the last two-tuple of TEK; go to step 7;

Step 7: Arrange the tags in TAGS in descending order according to the number of occurrences, take the top n of TAGS (n is an integer, you can take 1 or more according to needs), as the pre-classification tag of the TE, and go to step 8;

Step 8: Repeat steps 3 to 7 until the last TE pre-classification is completed; for Chinese webpages that fail to pre-classify, enter the classification calculation stage, and complete the classification of the training set after the calculation is completed, otherwise, directly end the pre-classification.

The specific steps of deduplicating the KWS of all training webpages described in step 2 and storing them in the keyword table KWT are as follows:

Step 2.1: Create a new mapping M, set each keyword K of the KWS of the first training web page TR as the key of M, and set the corresponding values to 1;

Step 2.2: For each keyword K in the KWS of the second TR, first judge whether K has been included in M, if it exists, add 1 to the value of the key-value pair whose key is K; if not, add <K,1> this key-value pair is added to M;

Step 2.3: Repeat step 2.2 for the remaining TRs until the last TR;

Step 2.4: Set a threshold s, when the value of the key-value pair is less than s, set the value to 0; otherwise, set the value of the key-value pair to 1;

The specific steps of calculating the binary group TEC of TEK described in step 4, that is, the keyword pair are as follows:

Step 4.1: There are N keywords K1, K2, ..., KN in TEK, search keyword pairs according to the following order, each keyword pair needs to enter step 4.2 for judgment: the keyword pair containing K1 is < K1,K2>,<K1,K3>,...,<K1,KN>, the keyword pair containing K2 is <K2,K3>,...,<K2,KN>, so until it contains K(N-1 ) keyword pair <K(N-1), KN>;

Step 4.2: If at least one of the current keyword pairs satisfies its corresponding value in M, go to step 5; if not, go back to step 4.1 and search for the next keyword pair.

In step 5, if the KWS of TR includes TEC, transfer the TAG of the TR to the TE, store it in the tag set TAGS of the TE, and count the number of occurrences of the tag at the same time. The specific steps are as follows:

Step 5.1: If the KWS of TR contains the first keyword of TEC, go to step 5.2; otherwise, go to step 6;

Step 5.2: If the KWS of TR contains the second keyword of TEC, add the TAG of TR to the tag set TAGS of TE. If the TAG is already included in the TAGS, add 1 to the value corresponding to the key-value pair with the TAG as the key; otherwise, add the <TAG, 1> key-value pair to the TAGS;

The reason why only two keywords need to be provided by TE in step 5.2 is that by counting the manually marked keywords, an appropriate threshold is set, and important keywords are obtained. If there is at least one important keyword in TE that appears in In TR, it is considered that the label of TR can be transferred to TE.

Example

Now take 7 categories with 200 training sets and 1000 test sets as an example to illustrate.

Mark a label for each training web page in the training set, and mark out the three keywords (as a keyword set) that best characterize its category, and store them in the memory.

The threshold setting for dividing important keywords is set to 0.2, and a keyword table containing 30 important keywords and 40 secondary keywords is obtained after frequency calculation.

For each test webpage in the test set, first traverse the keyword table to find out the keywords contained in it, the average case is 3 (excluding the test webpages that do not contain keywords, and the pre-classification of this part of the webpage fails), so there are at most 3 keyword pairs that meet the conditions.

For each two-tuple, the training set is traversed, and the tags of the training webpage containing the two-tuple in the keyword set are added to the label set of the test webpage. Finally, the label set is sorted according to the frequency, and the first two labels with the largest number of occurrences are taken as the labels of the test web page.

The method for pre-classifying Chinese webpages based on keyword matching in the present invention has been tested on real Chinese webpages. Compared with the classification results of Chinese webpages without pre-classification, the accuracy rate and recall rate of the final classification results are at least improved respectively. 10%, 15%, so that the classification effect of the entire Chinese webpage classification system has reached the expected value. The specific formula is as follows:

Recall rate (Recall) = related files retrieved by the system/total number of all related files in the system

Accuracy (Precision) = related files retrieved by the system/total number of files retrieved by the system

Original method: Recall=25%, Precision=18%;

The method in this paper: Recall=40%, Precision=28%.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (7)

1. The method for carrying out Chinese web page pre-classification based on keyword matching comprises the following steps: 1) Mark the category label TAG of each training webpage TR in the training set TRS and the keyword set KWS representing the category of the webpage, and generate a keyword table KWT with a key-value pair value of 1 in the mapping M; 2) extract the keywords contained in each test webpage TE in the test set TES according to the KWT, and form the keyword set TEK; 3) Calculate each binary group TEC of the TEK and traverse the training set, transfer the TAG of the TR containing the TEC in the KWS to the TE corresponding to the TEC, and store the TAG into the tag set TAGS of the TE, where The specific steps to calculate the binary TEC of TEK are as follows: 3-1-1) For a TEK containing N keywords K1, K2, ..., KN, search for keyword pairs in the following order, each keyword pair needs to enter step 3-1-2) for judgment : Keyword pairs containing K1 are <K1,K2>,<K1,K3>,...,<K1,KN>, keyword pairs containing K2 are <K2,K3>,...,<K2,KN> , until the keyword pair <K(N-1), KN> containing K(N-1) is reached; 3-1-2) If at least one of the current keyword pairs satisfies that its corresponding value in M is 1, traverse the training set TRS for TEC; if not, go back to step 3-1-1) and search next keyword pair; 4) Perform frequency statistics on the tags in TAGS, and take the tags with the highest frequency according to the requirements as the pre-classified tags of the test web page. 2. the method for carrying out Chinese webpage pre-classification based on keyword matching as claimed in claim 1, is characterized in that, in step 1), also comprises after the keyword set KWS of all training webpages belonging to the same category is deduplicated Generate keyword table KWT. 3. the method for carrying out Chinese web page pre-classification based on keyword matching as claimed in claim 1, is characterized in that, the concrete steps of generating keyword table KWT are as follows: 1-1) Create a new mapping M, use each keyword K of the KWS of the first training webpage TR as the key of M, and set the corresponding initial values to 1; 1-2) For each keyword K in the KWS of the second TR, first judge whether K has been included in M, if it exists, add 1 to the value of the key-value pair whose key is K; if it does not exist, then Add the key-value pair <K, 1> to M; 1-3) Repeat steps 1-2) for the remaining TRs until the last TR; 1-4) Set a threshold s, when the value of the key-value pair is less than s, set the value to 0; otherwise, set the value of the key-value pair to 1. 4. the method for carrying out Chinese web page pre-classification based on keyword matching as claimed in claim 1, is characterized in that, in step 3) in all binary groups TEC of TEK, at least one keyword is important keyword, and said important The keyword is the keyword with the highest frequency of occurrence. 5. the method for carrying out Chinese webpage pre-classification based on keyword matching as claimed in claim 4, is characterized in that, in step 2) also includes carrying out frequency statistics to the keyword that occurs in the test webpage, when secondary keyword occurs When the frequency exceeds the set threshold, it is marked as an important keyword. 6. the method for carrying out Chinese web page pre-classification based on keyword matching as claimed in claim 1, is characterized in that, step 3) in the process of each TEC traversal training set TRS specifically comprises: 3-2-1) If the KWS of TR contains the first keyword of TEC, go to step 3-2-2); otherwise, calculate the next TEC of TEK and start traversing the training set again; 3-2-2) If the KWS of TR contains at least one important keyword of TEC, and the important keyword is the keyword with the highest frequency of occurrence, add the TAG of TR to the tag set TAGS of TE, if TAGS already contains If the TAG is selected, add 1 to the value corresponding to the key-value pair with the TAG as the key; otherwise, add the <TAG,1> key-value pair to TAGS. 7. The method for pre-classifying Chinese webpages based on keyword matching as claimed in claim 1, wherein step 4) also includes performing classification calculations on test webpages TE that do not contain keywords.