CN1996316A - Search engine searching method based on web page correlation - Google Patents

Abstract

A search engine search method based on web page correlation. This method can provide two results to the user in one query process, effectively solves the problem of multiple words with one meaning and multiple meanings by using the information provided by the user's first click, and solves the problem that the keyword-based search engine cannot accurately determine To solve the problem of the user's query intention, this kind of webpage can not only provide the user with keywords related to the webpage related to the webpage of interest to the user, but also does not increase the complexity of the user's operation. In addition, using click data to update the difference matrix is to judge the difference between web pages from a new perspective. This difference is the difference in the statistical sense reflected in a large amount of data, and it is the difference in the process of using the search engine by a large number of search engine users. judgment made. Therefore, the present invention utilizes the web page-level correlation (difference) analysis with statistical stationarity, and can provide statistically optimized services for the user without long-term tracking of the user's behavior.

Description

A Search Engine Search Method Based on Web Page Correlation

technical field

The invention belongs to the technical field of search engine search in computer networks, in particular to a search engine search method based on web page correlation.

Background technique

Search engine technology is a technology that uses a combination of keywords to find relevant information on the Internet, sorts the information according to the matching degree of the keywords, and then returns it to the user for viewing. With the rapid development of the Internet, using search engines has become the most important way for network users to obtain network resources. In recent years, various search engines have emerged around the world, and these search engines have played an important role in the process of people obtaining information. At present, the main search engines can be divided into directory search engines and keyword-based search engines. Among them, the idea of the directory search engine is to pre-classify the webpage library, and then let the user choose which type of webpage he needs, and go to the corresponding directory to search. At present, the most representative classification directory search engine is yahoo[http: //www.yahoo.com]. However, in order to submit the best set of search results to users, it often requires very fine classification, and it is unrealistic for the existing manual and automatic classification techniques to be applied to massive network information. In addition, even if the search engine provides many If the classification is too fine, the user's selection process will become very complicated, and there is no guarantee that the user's judgment is completely consistent with the existing classification of the search engine.

Most of the search engines on the Internet currently use keyword-based query technology, and its typical representatives are Google [http://www.google.com] and Baidu [http://www.baidu.com].

The amount of information resources collected and indexed by this type of search engine through programs is extremely large, but most of the user's question sentences are composed of several words. Due to the ambiguity of the words themselves, it is difficult for the search engine to determine the user's needs. The situation will lead to a huge number of search results and the relevance cannot be guaranteed, so the user needs to spend a lot of effort to browse and filter the results of the search engine. In short, the quality of information given by the current search engines is not very high.

In addition, the sorting algorithms used by search engines generally include the following: (1) sorting algorithms based on word frequency statistics. The sorting algorithms used by many search engines in the early days were based on word frequency statistics. The calculation of word weight generally takes into account the position where the word appears in the web page. For example, words appearing in the title have higher weight than words in the text. However, due to the huge amount of network resources, the quality of two webpages with the same word frequency may be very different, and it is not reliable to calculate the correlation between webpages and keywords based on word frequency, so the limitations of this algorithm are obvious. (2) Sorting algorithm based on hyperlink analysis. The citation analysis method in traditional information retrieval theory is one of the important methods to determine the authority of academic literature, that is, to determine the authority of literature according to the quantity and quality of citations. The sorting algorithm based on hyperlink analysis draws on this idea. By applying the citation analysis idea to the calculation of the importance of network documents, the hyperlink structure of the network itself is used to assign the number of times the web page is cited and the importance of the cited web page itself. All web pages determine a level of importance to help optimize the ranking algorithm. However, this algorithm obtains the importance level of the webpage itself, rather than the relevance between the webpage and the keyword queried by the user, so it often happens that the webpage itself is of the highest quality in the query results, but it is not necessarily very relevant to the user's query needs. The problem.

Contents of the invention

In order to solve the above problems, the object of the present invention is to provide a webpage-based correlation algorithm that can accurately identify the user's needs without increasing the complexity of the operation, thereby improving the correlation between the search results of the search engine and the user's needs. Sexual search engine search method.

In order to achieve the above object, the search engine search method based on the web page correlation provided by the present invention comprises the following steps carried out in order:

(1) Record the click behavior data of network users on the search engine search result list for a period of time during the operation of the search engine;

(2) Calculate and save the degree of difference between all web pages with the method based on the vector space model;

(3) update the degree of difference between all web pages obtained in step 2 with the click data recorded in step 1;

(4) regard the degree of difference between the web pages obtained in step 3 as the distance between the web pages, and use the algorithm of dimension reduction to reduce the dimension of these distance data, thereby obtaining the low-dimensional geometric representation of the degree of difference data between the web pages;

(5) When the search engine receives a query request from a user, perform the following steps:

(a) The search engine accepts the query keyword input by the user, uses a correlation calculation method to obtain an initial query result list corresponding to the query keyword, and submits it to the user for viewing;

(b) the user will click on a link of interest after viewing the initial query list;

(c) The search engine records the first link clicked by the user, and marks the webpage corresponding to the link as the target webpage, and then calculates the target webpage and the initial query result according to the low-dimensional geometric representation of the difference data between webpages obtained in step 4 The degree of difference between the web pages corresponding to all the links in the list, and arrange the degree of difference in order from low to high to form a new query result;

(d) Submit a new query result to the user, which is the final query result that is related to the first web page clicked by the user and highly related to the query keyword entered by the user.

The recording time in the above step 1 takes every month as a cycle, and long-term dynamic tracking.

The search engine search method based on web page correlation provided by the present invention has the following beneficial effects:

1. The present invention can provide two results to the user in one query process, utilize the information provided by the user's first click to effectively solve the problem of multiple words with one meaning and one word with multiple meanings, and solve the problem that keyword-based search engines cannot The problem of accurately determining the user's query intention, this method of providing the second search result based on the user's first click can not only provide the user with keywords related to the web page that the user is interested in, but also does not increase user operations complexity.

2. From experience and intuition, only similar and highly relevant web pages are more likely to be accessed by users at the same time, so the click data includes the user's judgment on the difference of web pages. Using click data to update the difference matrix is to judge the difference between web pages from a new perspective. This difference is a statistical difference reflected in a large amount of data, which is made by a large number of search engine users in the process of using the search engine. judgment. Therefore, the present invention utilizes the web page-level correlation (difference) analysis with statistical stationarity, and can provide statistically optimized services for the user without long-term tracking of the user's behavior.

Detailed ways

The search engine search method based on webpage correlation provided by the present invention is to determine the type of information content that the user really needs by collecting the user's click behavior data, and at the same time, the click data is used as one of the basis for judging the correlation between webpages, thereby improving query efficiency. Relevance of results to user needs.

Usually, users who use search engines do not randomly click on links on the search result list, but make some kind of choice, so that the click data becomes a kind of implicit feedback that contains rich information. Since users are more inclined to click on links that match their needs, search engines can analyze the immediate needs of users by tracking the links clicked by users and solve the problem of ambiguity in query terms. For example, a search engine can provide a dynamic query result, so that the query result is not only related to the query word but also related to the content of the link that the user just clicked. need.

In the process of conducting a query, the user's needs are often relatively single, and generally they will not click for no reason, so the multiple links that are clicked during the user's query process are relatively related to each other. strong. The present invention saves the simultaneously clicked information through an n×n matrix as the basis for updating the correlation between web pages. That is to say, the present invention is to identify the query subject and query intention by tracking user clicks and web content difference information for each query request by maintaining the difference of webpage content obtained by a large number of user click data, and finally provide the user with a link with the user click. The final query results that are relevant to the first webpage of the website and are highly relevant to the query keywords entered by the user.

The search engine search method based on web page correlation provided by the present invention is described in detail below:

The search engine search method based on web page correlation provided by the present invention comprises the following steps carried out in order:

(1) Record the click behavior data of network users on the search engine search result list for a period of time during the operation of the search engine; since the click behavior data needs to be accumulated, this step needs to continue for a period of time with the search engine running.

(2) Use the method based on the vector space model to calculate and save the degree of difference between all web pages; the degree of difference between web pages is an attribute opposite to the degree of relevance of web pages, and it is a quantitative definition of the degree of difference between web pages. The higher the degree, the smaller the difference.

In this process, the difference matrix D is first established and updated to maintain the following data structure:

Common visit count matrix A: n*n symmetric matrix, which stores the counts of simultaneous visits among all web pages.

Click count vector B: n*1 vector, b _i is a non-negative integer, [0, +∞], each element stores the total number of clicks received by the corresponding web page.

Initial difference matrix D ⁰ : n*n symmetric matrix, calculated by vector space model. Let Doc={doc _i |1≤i≤n} denote a set of web pages. According to the vector space model, each webpage doc _i can be expressed as a vector doc _i , then the element d _ij ⁰ in row i and column j of D ⁰ can be defined as:

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}^{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; \&equiv;</span>\frac{{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\frac{{\mathrm{<span\; class="notranslate">\; doc</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; doc</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; doc</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; doc</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}{\mathrm{<span\; class="notranslate">\; max</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \{</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\frac{{\mathrm{<span\; class="notranslate">\; doc</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; doc</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; doc</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; doc</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \}</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

‖·‖ ₂ is the 2-norm. According to the definition, it can be seen that d _ij ⁰ is a normalized distribution in [0, 1] value, and the elements of D ⁰ satisfy the measure axiom (satisfying the measure axiom is a necessary attribute for D to obtain geometric embedding).

Click the difference matrix C: n*n matrix, and directly define the elements of C as

${\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; \&equiv;</span>\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; \{</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; j</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; j</span>}\end{array}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>\right.$

Difference matrix D: a symmetric matrix of n*n. The i-th row and j-th column element d _ij saves the difference between the i-th web page and the j-th web page, and d _ij is defined as

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; \&equiv;</span>\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}^{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; j</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; j</span>}\end{array}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>\right.$

Where w is a user parameter, 0<w<1. In the initial state w is set to 0, and the value of w is gradually increased as the system running time increases. After a long enough time, w can be 1. w can also be adjusted according to special needs. If some webpages only receive few clicks, the reliability of click data will be relatively low. At this time, w can be set to a smaller value, and the difference at this time mainly depends Values calculated by the VSM method.

The compressed representation of D is Y: n*d matrix, the compressed representation of D, and Y can be obtained by processing D with the dimensionality reduction algorithm. An element d _ij in D is denoted as the distance between the i-th and j-th row vectors in Y. Therefore, the difference between all web pages can be represented by the Euclidean distance of the vector in Y.

(3) Use the click data recorded in step 1 to update the degree of difference between all web pages obtained in step 2; the method for updating the degree of difference between any two web pages is as follows: (a) analyze the click data recorded in step 1, if the click data If it shows that these two webpages appear in a certain query result at the same time and they are both opened by the user at that time, the simultaneous click count between the two webpages will be increased by 1, and the two webpages can be obtained after processing all the click data in step 1. The total number of simultaneous hit counts between pages during the time period of step 1.

(4) The degree of difference between web pages obtained in step 3 is regarded as the distance between web pages, and the dimension reduction algorithm is used to reduce the dimension of these distance data, so as to obtain the low-dimensional geometric representation of the difference degree data between web pages; The data required by the engine to generate query results to calculate the degree of difference between web pages.

In the above steps 3 and 4, the difference matrix is regularly updated, and the update process is as follows

1. Generate an initial difference matrix D ⁰ according to the vector space model.

2. For each query event, generate a query result set according to a certain method (no need to restrict the specific algorithm used). The links in the result set are submitted to the user in order, and each link is accompanied by a summary of the corresponding web page.

3. After viewing the list, the user clicks several links according to the needs at that time, and the search engine records the clicked links and adds 1 to the simultaneous access count between the clicked web pages, as follows: For the clicked web pages i and j, execute

a _ij =a _ij +1 (4)

b _i =bi ₊₁ (5)

b _J =b _j +1 (6)

If only one webpage i is opened, execute

b _i =bi ₊₁ (7)

4. The search engine recalculates regularly according to A, B and D ⁰ to generate D, and reduces the dimension of D to obtain the compressed geometric representation Y of D. In this way, the difference between web pages is expressed as the Euclidean distance in the d-dimensional embedding space, d<<n.

5. When a new webpage is added, the system calculates the difference between the new webpage and other webpages with the method based on the vector space model, and adjusts the w parameter of the webpage to 0. When the number of clicks received by the web page reaches a certain amount, adjust w to a reasonable non-zero value.

(5) When the search engine receives a query request from a user, perform the following steps:

(a) The search engine accepts the query keyword input by the user, uses a certain correlation calculation method to obtain an initial query result list corresponding to the query keyword and submits it to the user for viewing;

(b) the user will click on a link of interest after viewing the initial query list;

(c) The search engine records the first link clicked by the user, and marks the webpage corresponding to the link as the target webpage, and then calculates the target webpage and the initial query result according to the low-dimensional geometric representation of the difference data between webpages obtained in step 4 The degree of difference between the web pages corresponding to all the links in the list, and arrange the degree of difference in order from low to high to form a new query result;

(d) Submit a new query result to the user, which is the final query result that is related to the first web page clicked by the user and highly related to the query keyword entered by the user.

In this step, when a user uses a search engine, the following process is performed for a query request:

1. Generate an initial query result set r using a method based on a vector space model. It is assumed that there are m web pages in r at this time.

2. After the user observes the initial query result and clicks on a link, the search engine records the link (called the target webpage, whose ID in the webpage database is i). Calculate the difference between the target webpage i and other webpages in r (that is, calculate the distance between the corresponding row vectors in Y), and obtain the difference vector $d_i\&equiv;[d_{{\mathrm{ij}}_1},{d_{\mathrm{ij}}}_2,...,{d_{\mathrm{ij}}}_m{]}^T$ (It is also possible to calculate the degree of difference between the target webpage and all other webpages and take a part of webpages with the smallest degree of difference as an extension of the query result set).

3. Arrange the web pages in r in ascending order according to the corresponding degree of difference in d _i , and submit them to the user. This is the final result submitted by the search engine to the user.

Claims (2)

1, a kind of search engine searching method based on web page correlation is characterized in that: described search engine searching method based on web page correlation comprises the following step that carries out in order:

(1) in the search engine operational process, writes down the click behavioral data of the network user in the search engine search results tabulation in a period of time;

(2) use method based on vector space model to calculate the diversity factor between all webpages and preserve;

(3) all webpage differences degree that obtain in the click data step of updating 2 with record in the step 1;

(4) the webpage differences degree that obtains in the step 3 is considered as distance between webpage, and the algorithm that subtracts approximately with dimension is to these range data dimensionality reductions, thereby obtains the low-dimensional geometric representation of webpage differences degrees of data;

(5) when receiving a user's one query request, search engine carries out the following step:

(a) search engine is accepted the key word of the inquiry of user input, draws one corresponding to initial query the results list of this key word of the inquiry and it is submitted to the user check with certain relatedness computation method;

(b) user will click its interested link after checking the initial query tabulation;

(c) first link of search engine recording user click, and the webpage that will link correspondence is designated as target web, the low-dimensional geometric representation of the webpage differences degrees of data that obtains according to step 4 calculates the diversity factor between the target web webpage corresponding with all-links in initial query the results list then, and diversity factor is constituted new Query Result according to from low to high series arrangement;

(d) new Query Result is submitted to the user, this Query Result promptly is the final Query Result of the relevant and key word of the inquiry height correlation that import with the user of first webpage of clicking with the user.

2, the search engine searching method based on web page correlation according to claim 1 is characterized in that: the writing time in the described step 1 with every month as the cycle, long-term dynamics is followed the tracks of.