CN112084150B - Model training, data retrieval method, device, equipment and storage medium - Google Patents

Abstract

The application discloses a model training and data retrieval method, a device, equipment and a storage medium, and relates to the fields of information retrieval and knowledge sharing. The specific implementation scheme is as follows: acquiring a historical search click log generated by a knowledge sharing system; generating a sample set according to the historical search click log; and training the model by using the sample set to obtain a target model. According to the implementation mode, the search click log generated by the user in the enterprise-level wiki is analyzed, and the target model is obtained through training, so that the required knowledge can be quickly and accurately found through the target model.

Description

Model training, data retrieval method, device, equipment and storage medium

Technical Field

The present application relates to the field of computer technology, specifically to the fields of information retrieval and knowledge sharing, and in particular to model training, data retrieval methods, devices, equipment and storage media.

Background technique

Such a problem is common in the development of large, medium and small enterprises. As the company grows, projects continue to accumulate, and employees continue to iterate, a large number of documents containing valuable experience and knowledge of employees are generated. If these documents are not managed online in a unified manner, it will be difficult to systematize and standardize the knowledge, and some knowledge and experience may be lost with the departure of key employees. Therefore, most companies have introduced enterprise-level wikis to centralize the knowledge documents accumulated in office scenarios in one location, which becomes the internal search engine of the company.

At the same time, a new problem is introduced: how to find the required knowledge quickly and accurately after having a huge amount of knowledge. Most enterprise-level wikis are very weak in meeting this user demand, which affects or even slows down the transfer of knowledge and office efficiency.

Summary of the invention

Provided are a model training, data retrieval method, device, equipment and storage medium.

According to the first aspect, a model training method is provided, including: obtaining a historical search click log generated by a knowledge sharing system; generating a sample set based on the historical search click log; and training a model using the sample set to obtain a target model.

According to a second aspect, a data retrieval method is provided, including: receiving a target search statement input by a user through a terminal; determining a target feature vector based on the target search statement and a target model as described in the first aspect; determining a feature vector for each document in a target search result for the target search statement; and sorting each document in the target result based on each feature vector and the target feature vector.

According to the third aspect, a model training device is provided, including: a log acquisition unit, configured to acquire historical search click logs generated by a knowledge sharing system; a sample generation unit, configured to generate a sample set based on the historical search click logs; and a model training unit, configured to train a model using the sample set to obtain a target model.

According to a fourth aspect, a data retrieval device is provided, including: a search statement receiving unit, configured to receive a target search statement input by a user through a terminal; a first vector determination unit, configured to determine a target feature vector based on the target search statement and the target model as described in the first aspect; a second vector determination unit, configured to determine a feature vector of each document in a target search result for the target search statement; and a document sorting unit, configured to sort each document in the target result based on each feature vector and the target feature vector.

According to the fifth aspect, a model training electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor is executed to enable the at least one processor to execute the method described in the first aspect.

According to the sixth aspect, a data retrieval electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor is executed by the at least one processor so that the at least one processor can execute the method described in the second aspect.

According to a seventh aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described in the first aspect.

According to an eighth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in the second aspect.

According to the technology of the present application, the search click logs generated by users in the enterprise-level wiki are analyzed and used to train a target model, through which the required knowledge can be found quickly and accurately.

It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

FIG1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

FIG2 is a flow chart of an embodiment of a model training method according to the present application;

FIG3 is a flow chart of another embodiment of a model training method according to the present application;

FIG4 is a flow chart of an embodiment of a data retrieval method according to the present application;

FIG5 is a schematic diagram of an application scenario of the model training method and data retrieval method according to the present application;

FIG6 is a schematic diagram of the structure of an embodiment of a model training device according to the present application;

FIG7 is a schematic structural diagram of an embodiment of a data retrieval device according to the present application;

FIG8 is a block diagram of an electronic device used to implement the model training method and data retrieval method of an embodiment of the present application.

Detailed ways

The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

FIG. 1 shows an exemplary system architecture 100 to which an embodiment of a model training method, a data retrieval method, a model training device or a data retrieval device of the present application can be applied.

As shown in Fig. 1, system architecture 100 may include terminal devices 101, 102, 103, network 104 and server 105. Network 104 is used to provide a medium for communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications, such as browser applications, social platform applications, etc., can be installed on terminal devices 101, 102, 103.

Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, e-book readers, car computers, laptop computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

The server 105 may be a server that provides various services, such as a background search server that searches for search statements sent by the terminal devices 101, 102, 103. The background search server may sort the search results and feed back the sorted search results to the terminal devices 101, 102, 103.

It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

It should be noted that the model training method and data retrieval method provided in the embodiments of the present application are generally executed by the server 105. Accordingly, the model training device and the data retrieval device are generally provided in the server 105.

It should be understood that the number of terminal devices, networks and servers in Figure 1 is only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

Continuing to refer to FIG2 , a process 200 of an embodiment of a model training method according to the present application is shown. The model training method of this embodiment includes the following steps:

Step 201, obtaining the historical search click log generated by the knowledge sharing system.

In this embodiment, the execution subject of the model training method (for example, it can be the server 105 shown in Figure 1, or it can be other electronic devices not shown in Figure 1) can obtain the historical search click log generated by the knowledge sharing system. Here, the knowledge sharing system can be an enterprise-level wiki, and employees within the enterprise can share knowledge through the knowledge sharing system, and can also use the knowledge sharing system to search and obtain knowledge by browsing the search results. During the search and browsing process, users will generate search click logs. Here, the execution subject can obtain the search click logs generated by the knowledge sharing system the previous day or the previous week as historical search click logs.

Step 202: Generate a sample set based on historical search click logs.

After obtaining the historical search click log, the execution subject can parse the historical search click log to obtain a sample set. Specifically, the execution subject can analyze the search statement input by the user from the historical search click log, and can also analyze the user's click information on each document in the search results. The execution subject can use the search statement and the clicked document as positive samples. The search statement and the unclicked document as negative samples. Thus, the execution subject can obtain multiple positive samples and multiple negative samples, which is a sample set.

Step 203: Use the sample set to train the model to obtain the target model.

After obtaining the sample set, the execution subject can use the sample set to train the model and obtain the target model. Specifically, the execution subject can use the search sentence in the sample set as input and the documents in the positive sample and the negative sample as the expected output to train the target model.

The model training provided by the above-mentioned embodiments of the present application analyzes the search click logs generated by users in the enterprise-level wiki, and uses the search click logs to train a target model, through which the required knowledge can be found quickly and accurately.

Continuing to refer to FIG3, it shows a process 300 of another embodiment of the model training method according to the present application. In the embodiment shown in FIG3, the above method may include the following steps:

Step 301, obtaining the historical search click log generated by the knowledge sharing system.

In this embodiment, the historical search click log includes at least one search statement and click information of each document in the search results of the at least one search statement. It is understandable that the historical search click log also includes the identification of the user who input each search statement, and can also include the information of each user's click.

Step 302: filter the historical search click log to obtain filtered data.

In this embodiment, the execution subject can filter the historical search click log to obtain the filtered data. In this way, the accuracy of the samples in the sample set can be improved, thereby improving the accuracy of the output results of the target model. Specifically, the execution subject can filter the search statements with a search count less than a preset value or the search statements with a search count greater than a preset value in the historical search click log.

In some optional implementations of this embodiment, the execution subject may filter the historical search click logs by the following steps:

Step 3021, determine the number of searches for each search statement and the number of clicks on each document based on the historical search click log.

Step 3022: filter the search statements whose search times are less than a first preset threshold and/or the documents whose click times are less than a second preset threshold to obtain filtered data.

In this implementation, the execution subject can first determine the number of searches for each search statement and the number of clicks on each document through the historical search click log. If the number of searches is too small, it means that the search statement is relatively remote and the learning effect is low, so it can be filtered. If the number of clicks is too small, it means that the document is not accurate for the search statement and can be filtered.

Step 303: for each search statement, according to the click information of each document in the search results for the search statement, a click document set for the search statement is determined from the search results.

In this embodiment, the execution entity can analyze each search statement included in the historical search click log to determine the click information of each document in the search results returned by each user for the search statement. In other words, the execution entity can analyze which document in the search results the user clicked on and which document the user did not click on, thereby obtaining a click document set.

Step 304: determine a positive sample set based on the search statement and the click document set.

The execution subject may determine the positive sample set by using the search statement and the obtained click document set. Specifically, the execution subject may use the search statement and a single click document as a single positive sample to obtain the positive sample set.

Step 305: Determine a negative sample set of the search statement based on the search results and the click document set.

After obtaining the clicked document set, the execution subject can also determine the unclicked documents according to the search results. The execution subject can use the search statement and the single unclicked document as a single negative sample to obtain a negative sample set.

In some optional implementations of this embodiment, the search results may include multiple documents and the ranking of each document. That is, each document has been ranked in the search results. The execution subject may obtain a negative sample set by following the steps below:

Step 3051: According to the above sorting, a plurality of unclicked documents adjacent to each clicked document in the clicked document set are determined to obtain the unclicked document set.

Step 3052: Obtain a negative sample set based on the search statement and the unclicked document set.

In this implementation, the execution subject can determine multiple unclicked documents adjacent to each clicked document in the above sorting to obtain a set of unclicked documents. For example, if the search statement is abc and the user clicks on the 4th document in the search results, then the documents in the 1st, 2nd, 3rd, 5th, and 6th items are unclicked documents. The execution subject can use the search statement and each unclicked document as negative samples to obtain a set of negative samples.

In some optional implementations of this embodiment, the ratio of the number of negative samples to the number of positive samples in the sample set is a preset value.

In this implementation, in order to ensure the learning ability of the model and to ensure that the amount of calculation during model training is not too large, the ratio of the number of negative samples to the number of positive samples can be controlled to be a preset value. The above preset value can be set according to the actual application scenario.

Step 306: For each sample in the training set, determine the feature vector corresponding to the document in the sample.

In this embodiment, the sample set may include a training set and a test set. Each sample in the training set is used to train the model, and each sample in the test set is used to test the trained model. For each sample in the training set, the execution subject may determine the feature vector corresponding to the document in the sample. Specifically, the execution subject may determine the feature vector corresponding to the document by using an existing feature extraction algorithm. Alternatively, the execution subject may determine the feature vector corresponding to the document by judging certain features of the document.

In some optional implementations of this embodiment, the execution entity may determine the feature vector corresponding to the document through the following steps: obtaining the feature vector based on the search sentence, the document and at least one pre-trained model in the sample.

In this implementation, the execution subject can obtain a feature vector based on the search sentence, document and at least one pre-trained model in the sample. For example, the execution subject can calculate at least one of the following information to determine the feature vector: the relevance score of the document obtained by inputting the search sentence and the document into the pre-trained relevance model respectively; the click weight of the document in the past week under the search sentence; the proportion of the document as the first point under the search sentence; the proportion of the document as the last point under the search sentence; the proportion of the document as a satisfactory click under the search sentence; the proportion of the document as a long click under the search sentence; the proportion of the document as a short click under the search sentence.

In the above information, the relevance model is used to calculate the relevance between the search statement and the document. The click weight can be calculated using the Wilson Score algorithm. This algorithm can be used for quality sorting. If the data includes both good and bad reviews, the algorithm can take into account the number of reviews and the rate of good reviews to calculate the score. The higher the score, the higher the quality of the data. For example, suppose doctor A has 100 reviews, 1 bad review and 99 good reviews, with a rate of 99%. Doctor B has 2 reviews, both good reviews, with a rate of 100%, which one should be ranked first? Using the Wilson algorithm, doctor A's score is 0.9440, doctor B's score is 0.3333, and doctor A ranks first.

In the above information, the first point refers to the first click made by the user when browsing the search results. The last point refers to the last click made by the user when browsing the search results. A satisfactory click refers to the last click under the last search statement under the same search intent. The so-called same search intent is determined based on the search statement. Specifically, the execution subject can determine whether the search statements are similar by calculating the similarity between the search statements and comparing the similarity with the preset threshold. If similar, the search intent is considered to be the same. When calculating the similarity between the search statements, it can be obtained by the edit distance between the search statements or by semantic analysis of the two. Long clicks can include satisfactory clicks. In addition, the last click of a non-last search statement under the same search intent is a long click if the time difference between the last click and the next search statement is greater than the first preset time length (e.g., 40s) and less than the second preset time length (e.g., 3600s). If the time difference between the last click and the next search statement is greater than the third preset time length (e.g., 0s) and less than the fourth preset time length (e.g., 5s), it is a short click. In addition, if the time difference between the previous click and the next click is greater than the first preset time length (e.g., 40s) and less than the second preset time length (e.g., 3600s) in two adjacent clicks, it is a long click. If the time difference between the previous click and the next click is greater than the third preset time length (e.g., 0s) and less than the fourth preset time length (e.g., 5s), it is a short click.

In addition, the execution entity may also determine at least one of the following information about the document: calculate the authority score of the document through a pre-trained authority model; calculate the quality score of the document through a pre-trained quality model; calculate the timeliness of the document (determined based on the current time, the creation time of the document, and a pre-set time period); calculate the authoritative characteristics of the document through the PageRank model (a web page ranking algorithm first proposed and used by Google. In essence, it is an algorithm that roughly analyzes the importance of web pages based on the number and quality of hyperlinks between web pages as the main factors); and count the search frequency of the search statement in the past week.

The execution entity may use the above information as the feature vector of the document in the sample.

Step 307: Take the search sentence in each sample in the training set as input, take the feature vector corresponding to the input search sentence as the expected output, and train to obtain the target model.

The execution subject can use the search sentences in each sample in the training set as the input of the model, and the feature vector corresponding to the input search sentence as the expected output, and train to obtain the target model.

Step 308: Use the test set to determine the search effect of the target model.

In this embodiment, the execution subject can also use the test set to determine the search effect of the target model. Specifically, the execution subject can use the search sentence in each sample of the test set as the input of the target model, and compare the output of the target model with the document corresponding to the input search sentence. It can be understood that if the two are similar, the search effect of the target model is considered to be good. If the two are not similar, the search effect of the target model is considered to be poor.

In some specific implementations, the execution subject can evaluate the effect of the target model through objective indicators and subjective indicators. Among them, the objective indicators include accuracy (ACC) and the area under the ROC curve and the coordinate axis (AUC). Subjective indicators can include GSB (good, same, bad).

The model training method provided in the above-mentioned embodiments of the present application can improve the accuracy of the model by performing a detailed analysis of the historical search click logs, filtering out invalid data therein, and using the remaining valid data to train the model.

Referring to FIG4 , it shows a process 400 of an embodiment of a data retrieval method according to the present application. As shown in FIG4 , the data retrieval method of the present embodiment may include the following steps:

Step 401: receiving a target search statement input by a user through a terminal.

In this embodiment, the execution subject of the data retrieval method (e.g., the server 105 shown in FIG. 1 ) can receive a target search statement input by a user through a terminal (e.g., the terminals 101, 102, 103 shown in FIG. 1 ). It should be noted that the execution subject of this embodiment can be the same as or different from the execution subject of the embodiments shown in FIG. 2 and FIG. 3 . The user can access the enterprise-level wiki through the terminal and input the target search statement through the enterprise-level wiki.

Step 402, determining a target feature vector according to the target search statement and the target model.

After receiving the target search statement, the execution subject may input the target search statement into the target model to obtain the target feature vector of the target search statement. Here, the target model may be the target model obtained in the embodiment of FIG. 2 or FIG. 3 .

Step 403: determine the feature vector of each document in the target search result of the target search statement.

The execution subject may further determine the feature vector of each document in the target search result of the target search statement. The execution subject may obtain the feature vector by analyzing each document. For example, by obtaining at least one of the following information of each document: obtaining the relevance score of the document by inputting the search statement and the document into the pre-trained relevance model respectively; the click weight of the document under the search statement in the past week; the proportion of the document as the first point under the search statement; the proportion of the document as the last point under the search statement; the proportion of the document as the satisfactory click under the search statement; the proportion of the document as the long click under the search statement; the proportion of the document as the short click under the search statement; calculating the authority score of the document through the pre-trained authority model; calculating the quality score of the document through the pre-trained quality model; calculating the timeliness of the document (determined according to the current time, the creation time of the document and the pre-set time period); calculating the authority feature of the document through the PageRank model (a web page ranking algorithm first proposed and used by Google. Essentially, it is an algorithm that roughly analyzes the importance of web pages based on the number and quality of hyperlinks between web pages as the main factors); and counting the search frequency of the search statement in the past week.

Step 404: sort the documents in the target result according to the feature vectors and the target feature vector.

The execution entity may sort the documents in the target result according to the feature vectors of each document and the target feature vector. Specifically, the execution entity may sort the documents corresponding to feature vectors similar to the target feature vector in the front part of the search results.

The data retrieval method provided by the above-mentioned embodiment of the present application can quickly and accurately retrieve relevant documents by utilizing a trained target model.

Continuing to refer to FIG5, it shows a schematic diagram of an application scenario of the model training method and data retrieval method according to the present application. In the application scenario shown in FIG5, the user accesses the enterprise-level wiki through the terminal 501 and inputs the search statement "abc" through it. The server 502 may include a target model locally, and after receiving the above search statement, multiple search results are obtained. Combined with the target model, each search result is sorted, and finally the sorting is output.

Further referring to FIG. 6 , as an implementation of the methods shown in the above figures, the present application provides an embodiment of a model training device, which corresponds to the method embodiment shown in FIG. 2 , and can be specifically applied to various electronic devices.

As shown in FIG. 6 , the model training device 600 of this embodiment includes: a log acquisition unit 601 , a sample generation unit 602 , and a model training unit 603 .

The log acquisition unit 601 is configured to acquire the historical search click log generated by the knowledge sharing system.

The sample generating unit 602 is configured to generate a sample set according to the historical search click log.

The model training unit 603 is configured to train the model using the sample set to obtain a target model.

In some optional implementations of this embodiment, the historical search click log includes at least one search statement and click information of each document in the search results for the at least one search statement. The sample generation unit 602 may be further configured to: for each search statement, determine a click document set of the search statement from the search results based on the click information of each document in the search results for the search statement; determine a positive sample set based on the search statement and the click document set; and determine a negative sample set of the search statement based on the search results and the click document set.

In some optional implementations of this embodiment, the search results include multiple documents and the ranking of each document. The sample generation unit 602 can be further configured to: determine multiple non-clicked documents adjacent to each clicked document in the clicked document set according to the ranking, and obtain a non-clicked document set; and obtain a negative sample set according to the search statement and the non-clicked document set.

In some optional implementations of this embodiment, the ratio of the number of negative samples to the number of positive samples in the sample set is a preset value.

In some optional implementations of this embodiment, the sample set includes a training set. The model training unit 603 can be further configured to: for each sample in the training set, determine the feature vector corresponding to the document in the sample; use the search sentence in each sample in the training set as input, use the feature vector corresponding to the input search sentence as the expected output, and train to obtain the target model.

In some optional implementations of this embodiment, the model training unit 603 may be further configured to obtain a feature vector based on the search sentence, document and at least one pre-trained model in the sample.

In some optional implementations of this embodiment, the device 600 may further include a data filtering unit not shown in FIG. 6 , which is configured to filter the historical search click log to obtain filtered data.

In some optional implementations of this embodiment, the data filtering unit is further configured to: determine the number of searches for each search statement and the number of clicks for each document based on historical search click logs; filter search statements whose search times are less than a first preset threshold and/or documents whose click times are less than a second preset threshold to obtain filtered data.

In some optional implementations of this embodiment, the sample set includes a test set. The apparatus 600 may further include a test unit (not shown in FIG. 6 ) configured to: determine the search effect of the target model using the test set.

It should be understood that units 601 to 603 recorded in the model training device 600 correspond to the steps in the method described in reference Figure 2. Therefore, the operations and features described above for the model training method are also applicable to the device 600 and the units contained therein, and will not be repeated here.

Further referring to FIG. 7 , as an implementation of the methods shown in the above figures, the present application provides an embodiment of a data retrieval device, which corresponds to the method embodiment shown in FIG. 4 , and can be specifically applied to various electronic devices.

As shown in FIG. 7 , the data retrieval device 700 of this embodiment includes: a search statement receiving unit 701 , a first vector determining unit 702 , a second vector determining unit 703 and a document sorting unit 704 .

The search sentence receiving unit 701 is configured to receive a target search sentence input by a user through a terminal.

The first vector determination unit 702 is configured to determine a target feature vector according to the target search statement and the target model described in the embodiments of FIG. 2 and FIG. 3 .

The second vector determination unit 703 is configured to determine a feature vector of each document in the target search result for the target search sentence.

The document ranking unit 704 is configured to rank each document in the target result according to each feature vector and the target feature vector.

It should be understood that units 701 to 704 recorded in the data retrieval device 700 correspond to the steps in the method described with reference to Figure 2. Therefore, the operations and features described above for the data retrieval method are also applicable to the device 700 and the units contained therein, and will not be repeated here.

According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

As shown in Figure 8, it is a block diagram of an electronic device according to an execution model training method and a data retrieval method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and/or required herein.

As shown in Figure 8, the electronic device includes: one or more processors 801, memory 802, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input/output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and/or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). In Figure 8, a processor 801 is taken as an example.

The memory 802 is the non-transient computer-readable storage medium provided in the present application. Among them, the above-mentioned memory 802 stores instructions executable by at least one processor to enable at least one processor to execute the execution model training method and data retrieval method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable the computer to execute the execution model training method and data retrieval method provided in the present application.

The memory 802, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions/modules corresponding to the model training method and the data retrieval method in the embodiments of the present application (for example, the log acquisition unit 601, the sample generation unit 602 and the model training unit 603 shown in FIG. 6, or the search statement receiving unit 701, the first vector determination unit 702, the second vector determination unit 703 and the document sorting unit 704 shown in FIG. 7). The processor 801 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 802, that is, implements the execution model training method and the data retrieval method in the above method embodiments.

The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of an electronic device that executes a model training method or a data retrieval method, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely disposed relative to the processor 801, and these remote memories may be connected to the electronic device that executes the model training method or the data retrieval method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The electronic device for executing the model training method and the data retrieval method may further include: an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803 and the output device 804 may be connected via a bus or other means, and FIG8 takes the bus connection as an example.

The input device 803 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device that executes the model training method and the data retrieval method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 804 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and/or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

According to the technical solution of the embodiment of the present application, the search click logs generated by users in the enterprise-level wiki are analyzed and used to train a target model, through which the required knowledge can be found quickly and accurately.

It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims (18)

1. A model training method, comprising:

Acquiring a historical search click log generated by a knowledge sharing system, wherein the historical search click log comprises at least one search statement and click information of each document in search results aiming at the at least one search statement;

For each search statement, determining a click document set of the search statement from search results of the search statement according to click information of each document in the search results; the search results include a plurality of documents and a ranking of the documents; determining a positive sample set according to the search statement and the click document set; determining a plurality of non-clicked documents adjacent to each clicked document in the clicked document set according to the sorting to obtain a non-clicked document set; obtaining a negative sample set according to the search statement and the non-clicked document set; the sample set comprises a training set;

For each sample of the training set, determining a feature vector corresponding to a document in the sample includes: calculating at least one of the following information to determine a feature vector: obtaining a relevance score of the document by respectively inputting a search sentence and a document into a pre-trained relevance model, wherein the click weight of the document in the last week under the search sentence, the proportion of the document as a head point under the search sentence, the proportion of the document as a tail point under the search sentence, the proportion of the document as a satisfactory click under the search sentence, the proportion of the document as a long click under the search sentence and the proportion of the document as a short click under the search sentence;

And taking the search sentences in each sample in the training set as input, taking the feature vectors corresponding to the input search sentences as expected output, and training to obtain a target model.

2. The method of claim 1, wherein a ratio of a number of negative samples to a number of positive samples in the sample set is a preset value.

3. The method of claim 1, wherein determining the feature vector corresponding to the document in the sample comprises:

The feature vector is derived based on the search term, the document, and at least one model pre-trained in the sample.

4. The method of claim 1, wherein the method further comprises:

and filtering the historical search click log to obtain filtered data.

5. The method of claim 4, wherein the filtering the historical search click log to obtain filtered data comprises:

determining the searching times of each searching sentence and the clicking times of each document according to the historical searching clicking logs;

Filtering the search sentences with the search times smaller than the first preset threshold and/or the documents with the click times smaller than the second preset threshold to obtain filtering data.

6. The method of claim 1, wherein the set of samples comprises a test set; and

The method further comprises the steps of:

And determining the searching effect of the target model by using the test set.

7. A data retrieval method comprising:

receiving a target search statement input by a user through a terminal;

Determining a target feature vector according to the target search statement and the target model according to claims 1-6;

determining feature vectors of all documents in target search results aiming at the target search statement;

And sorting all the documents in the target search result according to all the feature vectors and the target feature vector.

8. A model training apparatus comprising:

A log obtaining unit configured to obtain a history search click log generated by the knowledge sharing system, the history search click log including at least one search statement and click information of each document in search results for the at least one search statement;

A sample generating unit configured to determine, for each search term, a click document set of the search term from search results of the search term according to click information of each document in the search results; the search results include a plurality of documents and a ranking of the documents; determining a positive sample set according to the search statement and the click document set; determining a plurality of non-clicked documents adjacent to each clicked document in the clicked document set according to the sorting to obtain a non-clicked document set; obtaining a negative sample set according to the search statement and the non-clicked document set; the sample set comprises a training set;

a model training unit configured to determine, for each sample of the training set, a feature vector corresponding to a document in the sample, including: calculating at least one of the following information to determine a feature vector: obtaining a relevance score of the document by respectively inputting a search sentence and a document into a pre-trained relevance model, wherein the click weight of the document in the last week under the search sentence, the proportion of the document as a head point under the search sentence, the proportion of the document as a tail point under the search sentence, the proportion of the document as a satisfactory click under the search sentence, the proportion of the document as a long click under the search sentence and the proportion of the document as a short click under the search sentence; and taking the search sentences in each sample in the training set as input, taking the feature vectors corresponding to the input search sentences as expected output, and training to obtain a target model.

9. The apparatus of claim 8, wherein a ratio of a number of negative samples to a number of positive samples in the sample set is a preset value.

10. The apparatus of claim 8, wherein the model training unit is further configured to:

The feature vector is derived based on the search term, the document, and at least one model pre-trained in the sample.

11. The apparatus of claim 8, wherein the apparatus further comprises:

and the data filtering unit is configured to filter the historical search click log to obtain filtered data.

12. The apparatus of claim 11, wherein the data filtering unit is further configured to:

determining the searching times of each searching sentence and the clicking times of each document according to the historical searching clicking logs;

Filtering the search sentences with the search times smaller than the first preset threshold and/or the documents with the click times smaller than the second preset threshold to obtain filtering data.

13. The apparatus of claim 8, wherein the set of samples comprises a test set; and

The apparatus further comprises a test unit configured to:

And determining the searching effect of the target model by using the test set.

14. A data retrieval device comprising:

A search term receiving unit configured to receive a target search term input by a user through a terminal;

a first vector determination unit configured to determine a target feature vector from the target search statement and the target model of claims 1-6;

A second vector determination unit configured to determine feature vectors of respective documents in a target search result for the target search statement;

and the document ordering unit is configured to order the documents in the target search result according to the feature vectors and the target feature vector.

15. A model training electronic device, comprising:

At least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

16. A data retrieval electronic device, comprising:

At least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of claim 7.

17. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-6.

18. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of claim 7.