CN115186065A - Target word retrieval method and device - Google Patents

Abstract

The present application discloses a method and device for retrieving target words, which can be applied to the field of artificial intelligence. Obtaining the original retrieval phrase; determining a candidate set according to the original retrieval phrase; querying each query phrase in the candidate set, and determining the query result corresponding to each query phrase. The method extracts the latent semantic information contained in the retrieval database corpus by introducing latent semantics, and constructs a candidate set query with similar semantics based on the original query, so as to solve the problem of the lack of semantic matching ability of the existing full-text search, and improve the intelligence of the full-text search. user experience.

Description

Method and device for retrieving target words

technical field

The present application relates to the field of computer technology, and in particular, to a method and device for retrieving target words.

Background technique

Full-text search is a search to find out any content information stored in the entire book or article in the database. It can obtain information about chapters, sections, paragraphs, sentences, words and other information in the full text as needed, that is to say, it is similar to adding a label to each word in the whole book, and it can also perform various statistics and analysis. At present, the commonly used full-text search engines such as ES retrieval use the TFIDF (word frequency-reverse document frequency) algorithm to calculate the relevance at the bottom. This method is relatively limited. The TFIDF algorithm cannot mine the deep semantic relationship between words, which leads to the traditional ES search engine. Unable to deal with the situation of multiple words with one sense, stay at the low-level keyword retrieval level, and cannot provide users with semantic level retrieval. For example, if a user searches for "automobile", that is, a car, a traditional full-text search will only return records that contain the word "automobile", but records that actually contain the word "car" may also be what the user wants.

That is to say, in the current method for retrieving target words, the search results are usually limited to the literal text of the request sentence input by the user, and cannot deeply capture the real intention behind the sentence input by the user. The retrieval method of such words has poor recall and low precision.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present application provide a method and apparatus for retrieving target words, aiming at realizing accurate full-text retrieval of target words.

In a first aspect, an embodiment of the present application provides a method and device for retrieving target words, the method comprising:

Get the original search phrase;

A candidate set is determined according to the original retrieval phrase; the candidate set includes an original retrieval phrase and a plurality of first retrieval phrases, and the first retrieval phrase is a phrase semantically similar to the original retrieval phrase

Each query phrase in the candidate set is queried, and a query result corresponding to each query phrase is determined.

Optionally, the determining the candidate set according to the original search phrase includes:

Obtain latent semantic computing models;

A plurality of first search phrases are determined by using the latent semantic calculation model and the first rule, and the first search phrases are phrases that are semantically similar to the original search phrases; the first rules are used to determine the first search phrases the number of phrases;

The original search phrase and the plurality of first search phrases are combined to form a candidate set.

Optionally, the determining the query result corresponding to each query phrase includes:

determining a text search record according to the query phrase, the text search record including text-related words;

The query result is determined according to a second rule, and the second rule is used to determine the number of the text-related words in the query result.

Optionally, after determining the query result corresponding to each query phrase, the method further includes:

The query results corresponding to each query phrase are combined, and the combined result is used as a final query result set.

Optionally, the latent semantic computing model is an implicit semantic analysis model or a word vector model.

In a second aspect, an embodiment of the present application provides an apparatus for retrieving target words, the apparatus comprising:

The original retrieval phrase acquisition module is used to obtain the original retrieval phrase;

A candidate set determination module, configured to determine a candidate set according to the original retrieval phrase; the candidate set includes an original retrieval phrase and a plurality of first retrieval phrases, and the first retrieval phrase is a phrase semantically similar to the original retrieval phrase ;

The query result determination module is configured to query each query phrase in the candidate set, and determine the query result corresponding to each query phrase.

Optionally, the candidate set determination module includes:

A computational model obtaining module, used to obtain a latent semantic computational model;

A first retrieval phrase determination module, configured to determine a plurality of first retrieval phrases through the latent semantic computing model and the first rule, where the first retrieval phrases are phrases semantically similar to the original retrieval phrases; the first retrieval phrases are semantically similar to the original retrieval phrases; rules are used to determine the number of the first search phrases;

A candidate set forming module is configured to combine the original search phrase and the plurality of first search phrases to form a candidate set.

Optionally, the query result determination module includes:

a text search record determination module, configured to determine a text search record according to the query phrase, where the text search record includes text-related words;

The query result determination module is configured to determine the query result according to a second rule, and the second rule is used to determine the number of the text-related words in the query result.

Optionally, the device further includes:

The merging module is used for merging the query results corresponding to each query phrase, and using the merged result as the final query result set.

Optionally, the latent semantic computing model is an implicit semantic analysis model or a word vector model.

Embodiments of the present application provide a method and device for retrieving target words. When executing the method, an original search phrase is obtained; a candidate set is determined according to the original search phrase; each query phrase in the candidate set is queried, and a query result corresponding to each query phrase is determined. Thus, after the user inputs the target retrieval phrase, the system analyzes the target retrieval phrase to obtain synonymous phrases, and uses the original text to be retrieved and multiple semantically similar text records as retrieval conditions to obtain multiple retrieval results. In this way, the comprehensive retrieval effect of the target word is achieved. In this way, the retrieval results integrate keyword retrieval and semantic retrieval, which improves the recall and precision of existing full-text retrieval.

Description of drawings

In order to more clearly illustrate the technical solutions in the present embodiment or the prior art, the following briefly introduces the accompanying drawings required in the description of the embodiment or the prior art. Obviously, the accompanying drawings in the following description are only For some embodiments of the present application, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a flowchart of a method for retrieving target words provided by an embodiment of the present application;

2 is a flowchart of a method for retrieving a target word provided by an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a target word retrieval provided by an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

As mentioned above, currently commonly used full-text search engines, such as ES, use the TFIDF algorithm to calculate the relevance at the bottom. However, the inventor found through research that such methods are relatively limited. The TFIDF algorithm cannot dig out the deep semantic relationship between words, so this retrieval method cannot deeply capture the real intention behind the sentence input by the user. It has the disadvantages of poor recall and low accuracy.

In order to solve this problem, an embodiment of the present application provides a method and device for retrieving target words. When the method is executed, an original search phrase is obtained; a candidate set is determined according to the original search phrase; and a candidate set is queried for For each query phrase, a query result corresponding to each query phrase is determined. Thus, after the user inputs the target retrieval phrase, the system analyzes the target retrieval phrase to obtain synonymous phrases, and uses the original text to be retrieved and multiple semantically similar text records as retrieval conditions to obtain multiple retrieval results. In this way, the comprehensive retrieval effect of the target word is achieved. In this way, the retrieval results integrate keyword retrieval and semantic retrieval, which improves the recall and precision of existing full-text retrieval.

The methods provided by the embodiments of the present application are executed by a search engine and a background server. For example, the background server includes a retrieval system with a retrieval function and an integration function. After the retrieval system obtains the semantic computing model in the retrieval database, it analyzes the original retrieval phrases to obtain phrases with similar semantics. The background server may be a server device or a server cluster composed of multiple servers.

The following describes the method for retrieving target words provided by the present application through an embodiment. Please refer to FIG. 1. FIG. 1 is a flowchart of a method for retrieving target words provided by an embodiment of the present application, including:

S101: Obtain the original search phrase.

The original retrieval phrase is the initial retrieval target phrase. In a specific application scenario, the original retrieval phrase may be input by the user, or may be set by the system according to query requirements.

S102: Determine a candidate set according to the original search phrase.

The candidate set includes an original retrieval phrase and a plurality of first retrieval phrases, and the first retrieval phrases are phrases that are semantically similar to the original retrieval phrases.

Assuming that the original search phrase of the query is "big data research", the first two records with the most similar semantics of the phrase to be searched in the search database are calculated by the latent semantic model as "investigation" and "learning", then these two records are the first search phrase. The system can include the original search phrases obtained by the query and a plurality of first search phrases into the candidate set. In practical application scenarios, the system can set different flags for each first retrieval phrase according to the different relevancy degrees between different first retrieval phrases and the original retrieval phrase, which can be used to distinguish retrieval results with different similarities in the subsequent process.

For details on how to determine the candidate set according to the original search phrase, please refer to the following paragraphs, which will not be repeated here.

S103: Query each query phrase in the candidate set, and determine a query result corresponding to each query phrase.

The query text is the phrases that are queried one by one in the candidate set. The original retrieval phrase and a plurality of first retrieval phrases are sequentially taken out from the candidate set, and used as query phrases one by one. Determine the query result corresponding to each query phrase.

In practical application scenarios, the system can combine the query results corresponding to each query phrase to form a final query result set.

The method for retrieving target words provided by the embodiments of the present application will be described in detail below. Referring to FIG. 2 , FIG. 2 is another schematic flowchart of the retrieval of target words passed in this embodiment of the present application. The specific process is as follows:

S201: Obtain the original search phrase.

The system obtains the original search phrases that need to be searched.

S202: Obtain a latent semantic computing model.

Obtain the corpus in the retrieval database to calculate the latent semantic model offline. The model can be an LSA model or a Word2vec model, which is not limited here. The following steps are implemented using LSA as an example, including:

Analyze the collection of documents to build a lexical-text matrix A. Suppose A is an m*n text data matrix (n<<m), which means that the corpus contains m words and n documents.

Perform singular value decomposition on the vocabulary-text matrix, reduce the dimension of the matrix after SVD decomposition, and use the reduced matrix to build the LSA model of the latent semantic space. The formula is:

为话题文本矩阵，A可以分解出k个特征值，k在此处可以代指主题数，我们选取排序后其中较大的r个特征值，r的值可以根据下列公式计算得出：In the formula, A _m×n is the m*n text data matrix, which decomposes the large matrix A into truncated singular value decomposition and decomposes it into a product matrix of 3 matrices, U _m×k is the word-topic matrix,

For the topic text matrix, A can be decomposed into k eigenvalues, where k can refer to the number of topics, we select the larger r eigenvalues after sorting, and the value of r can be calculated according to the following formula:

In the formula, P _r is the square sum of the first r larger eigenvalues of the diagonal matrix, and P is the square sum of all the eigenvalues of the diagonal matrix. The calculated r can have more than 95% of the information of the original matrix, and r is far from less than k.

就可以近似表示矩阵A。U是单词-话题矩阵，每一列代表一个潜语义，这个潜语义的意义由m个单词按不同权重组合而成，其行代表词语，其列代表文档。一般情况下，词-文档矩阵的元素是该词在文档中的出现次数。因为U中每一列相互独立，所以r个潜语义构成了一个语义空间，矩阵U中的每一列表示一个关键词，数值越大则越相关，因此，通过U_m×r可以看到词与词义之间的相关性。矩阵V中的每一行表示一类主题，其中的每个非零元素表示一个主题与一个文档的相关性，从

可以看出文本与主题的相关性。而∑V^T是话题-文档矩阵，∑V^T中每一列代表一篇文档，将该文档被映射到了语义空间，∑中每一个奇异值指示了该潜语义的重要度，矩阵∑表示文章主题和关键词之间的相关性。so

The matrix A can be approximated. U is a word-topic matrix, and each column represents a latent meaning. The meaning of this latent meaning is composed of m words combined with different weights. Its rows represent words and its columns represent documents. In general, the elements of the word-document matrix are the number of occurrences of the word in the document. Because each column in U is independent of each other, r subliminal semantics constitute a semantic space. Each column in the matrix U represents a keyword, and the larger the value, the more related it is. Therefore, through U _m×r , you can see the word and the meaning of the word correlation between. Each row in matrix V represents a class of topics, where each non-zero element represents the relevance of a topic to a document, starting from

The relevance of the text to the topic can be seen. ∑V ^T is a topic-document matrix, each column in ∑V ^T represents a document, and the document is mapped to the semantic space, each singular value in ∑ indicates the importance of the latent semantics, and the matrix ∑ represents the topic of the article and keywords.

S203: Determine a plurality of first search phrases by using the latent semantic computing model and the first rule.

The first rule is used to determine the number of the first search phrases. In a specific application scenario, the first rule may be set by the user, or may be set by the system according to query requirements.

In some possible implementations, the user can select or delete the search records with similar semantics provided by the system according to personal search requirements, or the user can select different first retrieval phrases and original retrieval phrases to form different combinations according to requirements . For example, when the user deletes the search phrase with the second semantic similarity, the search phrase with the third semantic similarity can be automatically replaced. The first retrieval phrase can also be set by the user, and other retrieval phrases with similar semantics obtained by the system retrieval are selected or the content of the first retrieval phrase can be input by themselves.

For example, when the first rule indicates that the number of the first search phrases is 2, in an actual application scenario, the system uses the latent semantic model to calculate the top two records with the most similar semantics in the search database for the phrases to be searched. Assuming that the query is: query="big data research", the top 2 records with the most similar semantics of the phrase to be retrieved in the retrieval database are calculated by the latent semantic model as: similar_top_2={"hadoop research", "spark learning"}. Among them, "investigation" and "learning" are the first search phrases corresponding to the current first rule.

Among them, both Hadoop and Spark are representative signs of different first retrieval phrases. In the actual application process, the system can distinguish the first search phrases with different representative marks having different degrees of similarity. In some possible implementations, the system can set and adapt the representative marks of the first search phrase.

In practical applications, both Hadoop and Spark are big data frameworks, and Spark is a fast and general computing engine designed for large-scale data processing. Hadoop is a distributed system infrastructure developed by the Apache Foundation. Hadoop processes data in a reliable, efficient, and scalable manner. In the application process, the data retrieved by the system can be further processed according to these two big data frameworks.

As a further optimization, in S203, the latent semantic model is used to calculate the top N records with the most similar semantics of the phrase to be retrieved in the retrieval database, specifically including:

For a given query, we construct a pseudo-document from the words A _q contained in this query: V _q =A _q U∑ ^-1 , and then calculate the cosine similarity between the pseudo-document and each column in V to get the sum given by Query the most similar N documents. Assuming that the text vector corresponding to the t-th column in V is V _t , the formula for calculating the cosine similarity between the pseudo-document vectors V _q and V _t is:

where V _q and V _t are the vector representations of the query text and the text corresponding to the t-th column in the semantic space matrix, ||V _q || and ||V _t || are the modulo of the vectors V _q and V _t , cos (V _q , V _t ) is the cosine similarity between the text vector and the document vector.

S204: Combine the original search phrase and the plurality of first search phrases to form a candidate set.

Merge the original search phrase and multiple first search phrases corresponding to the first rule to form a candidate set. For example, combine the original search phrase "big data research" with the first search phrases "hadoop research" and "spark learning" to obtain candidate_list= {"big data research", "hadoop research", "spark learning"}.

S205: Query each query phrase in the candidate set, and determine a text search record according to the query phrase.

Wherein, the text search record includes a plurality of text-related words, and the text-related words are words related to the query phrase obtained from the query. A text search record is a collection of related terms.

Specifically, the system sequentially retrieves the phrases to be queried from the candidate set, and determines the text search record corresponding to the current phrase according to the query phrase. The text search records can be "Research on the Status Quo of Big Data Development", "Research on Big Data Related Components", "Relationship between Big Data and Artificial Intelligence", "Exploration of Big Data Development Prospects", and "Application Development of Big Data". The full-text search result of the current query text, that is, if there are N contents directly related to the current query phrase in the full text, the number of text search records may be N.

In some possible implementations, the system may sort each related term based on the level of relevancy to the query phrase. That is, in the process of acquiring text-related phrases, the system sorts the acquired text-related phrases according to the degree of relevancy to form sequential text search records. Therefore, in the current text search records, there is a correlation order relationship between multiple text-related words. Taking the query phrase big data research as the correlation judgment criterion, the correlation degree of the research on the development status of big data is higher than that of the research on the related components of big data. The degree gradually decreases backwards.

In some possible implementations, the system retrieves the full text only according to the query phrase, and does not rank the obtained related words during the retrieval process. Therefore, there is no order of correlation between each text-related word in the formed text search record. Regarding the extraction of text-related words in the text search records according to the degree of relevancy, after obtaining the number request corresponding to the second rule, the system can sort the text-related words in the text search records, or set the degree of relevancy threshold directly to the text. A plurality of text-related words in the search record are filtered to obtain the number of text-related words corresponding to the second rule. That is, the system selects a number of text-related phrases whose relevance reaches the standard from the set of text search records with no order relation.

S206: Determine the query result according to the second rule.

The second rule is used to determine the number of the text-related words in the query result. In a specific application scenario, the second rule may be set by the user, or may be set by the system according to query requirements.

For example, when the second rule indicates that the number of text-related phrases is 3, in an actual application scenario, the system takes the top 3 text records of relevance to form the query result. Assuming that the text search records corresponding to the current query phrase are "Research on the current situation of big data development", "Research on big data related components", "Relationship between big data and artificial intelligence", "Exploration on the development prospect of big data", "Application expansion of big data", According to the text-related phrase selection rule mentioned in step S205, select the top 3 text records of relevance to form the query result, then the corresponding result set result1={"Research on the current situation of big data development", "Research on big data related components", " The relationship between big data and artificial intelligence”}.

S207: Combine the query results corresponding to each query phrase, and use the combined result as a final query result set.

According to the above step S206, the query results corresponding to several query texts in the candidate set are determined.

For example, each query text is sequentially retrieved from the candidate set and input into a full-text search engine such as ES for retrieval, and the top 3 text records of relevance are retrieved to form query results. Query text query1="big data research", query2="hadoop research", query3="spark learning", corresponding result set result1={"big data development status research", "research on big data related components", "big data" Relationship with artificial intelligence"}, result2={"hadoop research", "hadoop technology research", "hadoop quick start"}, result3={"spark learning", "spark study notes", "spark basic tutorial"}. In the current step, the query results corresponding to each query phrase are combined to form a retrieval result set: result_list={"Research on the current situation of big data development", "Research on components related to big data", "The relationship between big data and artificial intelligence", "hadoop Research", "hadoop technology research", "hadoop quick start", "spark learning", "spark study notes", "spark basic tutorial"}.

In practical application scenarios, the query results can be classified and displayed on the user-side search interface. For example, the original search phrase and the first search phrase with different similarities can be distinguished, and the query result of the original search phrase can be set in the first row, and then , the similarity of the query result corresponding to the first search phrase decreases with the number of rows. Alternatively, the system can set different colors for different phrases, thereby achieving a distinguishing effect on the display interface.

The above embodiments of the present application provide some specific implementations of a retrieval method based on latent semantic analysis, and based on this, the present application also provides a corresponding device. The device provided by the embodiments of the present application will be introduced below from the perspective of functional modularity.

Please refer to FIG. 3 , which is a schematic structural diagram of an apparatus for retrieving a target word according to an embodiment of the present application.

In this embodiment, the device may include:

an original retrieval phrase acquisition module 300, configured to acquire an original retrieval phrase;

A candidate set determination module 301, configured to determine a candidate set according to the original retrieval phrase; the candidate set includes an original retrieval phrase and a plurality of first retrieval phrases, and the first retrieval phrase is semantically similar to the original retrieval phrase phrase;

The query result determination module 302 is configured to query each query phrase in the candidate set from the target text, and determine the query result corresponding to each query phrase.

Optionally, the candidate set determination module includes:

A computational model obtaining module, used to obtain a latent semantic computational model;

A first retrieval phrase determination module, configured to determine a plurality of first retrieval phrases through the latent semantic computing model and the first rule, where the first retrieval phrases are phrases semantically similar to the original retrieval phrases; the first retrieval phrases are semantically similar to the original retrieval phrases; rules are used to determine the number of the first search phrases;

A candidate set forming module is configured to combine the original search phrase and the plurality of first search phrases to form a candidate set.

Optionally, the query result determination module includes:

a text search record determination module, configured to determine a text search record according to the query phrase, where the text search record includes text-related words;

The query result determination module is configured to determine the query result according to a second rule, and the second rule is used to determine the number of the text-related words in the query result.

Optionally, the device further includes:

The merging module is used for merging the query results corresponding to each query phrase, and using the merged result as the final query result set.

Optionally, the device further includes:

The latent semantic computing model is a latent semantic analysis model or a word vector model.

It should be noted that the method and device for retrieving target words provided by the present invention can be used in the field of artificial intelligence. The above is only an example, and does not limit the application field of the method and device for retrieving a target word provided by the present invention.

The method and device for retrieving a target word provided by the present application have been described in detail above. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method. It should be pointed out that for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can also be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities or operations. There is no such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection of the present application. within the range.

Claims (10)

1. A method for retrieving a target word, the method comprising:

acquiring an original retrieval phrase;

determining a candidate set according to the original retrieval phrase; the candidate set comprises an original search phrase and a plurality of first search phrases which are semantically similar to the original search phrase

And querying each query phrase in the candidate set, and determining a query result corresponding to each query phrase.

2. The method of claim 1, wherein said determining a candidate set from said original search phrase comprises:

acquiring a potential semantic computation model;

determining a plurality of first search terms through the latent semantic calculation model and a first rule, wherein the first search terms are terms similar to the original search terms in semantic meaning; the first rule is used for determining the number of the first search phrase;

merging the original search phrase with the plurality of first search phrases to form a candidate set.

3. The method for retrieving the target term according to claim 1, wherein the determining the query result corresponding to each query phrase comprises:

determining a text search record according to the query phrase, wherein the text search record comprises text related terms;

and determining the query result according to a second rule, wherein the second rule is used for determining the number of the text related terms in the query result.

4. The method for retrieving the target term in claim 1, wherein after determining the query result corresponding to each query phrase, the method further comprises:

and combining the query results corresponding to each query phrase, and taking the combined results as a final query result set.

5. The method for retrieving target words according to claim 2, wherein the latent semantic calculation model is an implicit semantic analysis model or a word vector model.

6. An apparatus for retrieving a target word, the apparatus comprising:

the original retrieval phrase acquisition module is used for acquiring an original retrieval phrase;

a candidate set determining module, configured to determine a candidate set according to the original search phrase; the candidate set comprises an original search phrase and a plurality of first search phrases, wherein the first search phrases are phrases which are similar to the original search phrase in semantic meaning;

and the query result determining module is used for querying each query phrase in the candidate set and determining a query result corresponding to each query phrase.

7. The apparatus of claim 6, wherein the candidate set determination module comprises:

the calculation model acquisition module is used for acquiring a potential semantic calculation model;

a first search phrase determination module, configured to determine a plurality of first search phrases according to the latent semantic calculation model and a first rule, where the first search phrases are phrases semantically similar to the original search phrase; the first rule is used for determining the number of the first search phrase;

and the candidate set forming module is used for combining the original search phrase and the plurality of first search phrases to form a candidate set.

8. The apparatus of claim 6, wherein the query result determination module comprises:

the text search record determining module is used for determining text search records according to the query phrases, wherein the text search records comprise text related terms;

and the query result determining module is used for determining the query result according to a second rule, and the second rule is used for determining the number of the text related terms in the query result.

9. The apparatus of claim 6, further comprising:

and the merging module is used for merging the query results corresponding to each query phrase, and taking the merged results as a final query result set.

10. The apparatus of claim 7, wherein the latent semantic computation model is an implicit semantic analysis model or a word vector model.

Images