CN118656452A - Text evaluation model training method, text evaluation method and related device - Google Patents

Abstract

The application relates to a training method of a text evaluation model, a text evaluation method and a related device. The method comprises the following steps: determining candidate simulation objects of the target object; determining a matching simulation object matched with the target object from the candidate simulation objects; establishing a session between a target object and a matched simulation object, and acquiring a session text of the matched simulation object and a text label of the session text in the session process; based on the session text and the text labels, training the text evaluation model. Acquiring a text to be evaluated of a target object in a conversation process; performing speech evaluation processing on the text to be evaluated through the text evaluation model, and determining a text evaluation result of the text to be evaluated; determining whether text to be evaluated needs text optimization or not based on a text evaluation result; and under the condition that text optimization is required, text optimization processing is carried out on the text to be evaluated, and optimized text is obtained. By adopting the method, the text evaluation can be accurately performed.

Description

Text evaluation model training method, text evaluation method and related device

Technical Field

The present application relates to the field of artificial intelligence technology, and in particular to a text evaluation model training method, a text evaluation method and related devices.

Background Art

Communication between people requires certain skills. Taking the conversation between users and customer service agents as an example, the skills of customer service agents refer to the ability of customer service agents to communicate with users and solve user problems, which is the most important factor in the work of customer service agents. Therefore, it is necessary to improve the skills of customer service in conversation to a certain extent to improve the quality of related products or services, thereby bringing economic benefits. Before improving the conversation skills of customer service, it is necessary to evaluate whether the conversation skills need to be improved. Generally, the conversation text of customer service is evaluated to determine whether the conversation skills need to be improved or the conversation text needs to be optimized. Therefore, how to train an accurate text evaluation model has become a hot research issue in this field.

Summary of the invention

Based on this, it is necessary to provide an accurate text evaluation model training method, a text evaluation method and related devices to address the above technical problems.

In a first aspect, the present application provides a method for training a text evaluation model, the method comprising:

Determine candidate simulation objects of the target object; the candidate simulation objects include shadow simulation objects of the target object and shadow simulation objects of non-target objects;

Determine a matching simulation object that matches the target object from the candidate simulation objects;

Establishing a conversation between the target object and the matching simulation object, and obtaining a conversation text of the matching simulation object and a text label of the conversation text during the conversation;

A text evaluation model is trained based on the conversation text and the text label.

In a second aspect, the present application also provides a training device for a text evaluation model, the device comprising:

A determination module, configured to determine candidate simulation objects of a target object; the candidate simulation objects include a shadow simulation object of the target object and a shadow simulation object of a non-target object;

A matching module, used to determine a matching simulation object that matches the target object from the candidate simulation objects;

A conversation module, used to establish a conversation between the target object and the matching simulation object, and obtain a conversation text of the matching simulation object and a text label of the conversation text during the conversation;

A training module is used to train a text evaluation model based on the conversation text and the text label.

In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

Determine candidate simulation objects of the target object; the candidate simulation objects include shadow simulation objects of the target object and shadow simulation objects of non-target objects;

Determine a matching simulation object that matches the target object from the candidate simulation objects;

Establishing a conversation between the target object and the matching simulation object, and obtaining a conversation text of the matching simulation object and a text label of the conversation text during the conversation;

A text evaluation model is trained based on the conversation text and the text label.

In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

Determine candidate simulation objects of the target object; the candidate simulation objects include shadow simulation objects of the target object and shadow simulation objects of non-target objects;

Determine a matching simulation object that matches the target object from the candidate simulation objects;

Establishing a conversation between the target object and the matching simulation object, and obtaining a conversation text of the matching simulation object and a text label of the conversation text during the conversation;

A text evaluation model is trained based on the conversation text and the text label.

In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

Determine candidate simulation objects of the target object; the candidate simulation objects include shadow simulation objects of the target object and shadow simulation objects of non-target objects;

Determine a matching simulation object that matches the target object from the candidate simulation objects;

Establishing a conversation between the target object and the matching simulation object, and obtaining a conversation text of the matching simulation object and a text label of the conversation text during the conversation;

A text evaluation model is trained based on the conversation text and the text label.

The training method, text evaluation method and related device of the above-mentioned text evaluation model, firstly, obtain the conversation text and text label by analyzing the conversation process between the target object and the candidate simulation object, and use the conversation text and text label to train the text evaluation model, and the trained text evaluation model can support accurate evaluation of the text to be evaluated. Among them, the candidate simulation object includes the shadow simulation object of the target object and the shadow simulation object of the non-target object, and can use different training data to comprehensively train the model, so that the multiple text evaluation models obtained by training can be used to perform comprehensive text evaluation processing on the text to be evaluated, thereby improving the accuracy of the text evaluation. Further, by determining the shadow simulation object of the target object and the shadow simulation object of the non-target object, the target object can use the method of conversation with the simulation object that is the same as or different from its own expression habits, respectively, to achieve self-reflection and self-learning of the target object, improve the conversation level and evaluation ability of the target object, and then improve the accuracy of the text label used for model training, thereby improving the accuracy of the text evaluation model.

In a sixth aspect, the present application also provides a text evaluation method, the method comprising:

Get the text to be evaluated of the target object during the conversation;

Performing speech evaluation processing on the text to be evaluated by a text evaluation model to determine a text evaluation result of the text to be evaluated; wherein the text evaluation model is trained using the training method of the text evaluation model;

Based on the text evaluation result, determining whether the text to be evaluated needs to be optimized;

When text optimization is required, text optimization processing is performed on the text to be evaluated to obtain an optimized text.

In a seventh aspect, the present application further provides a text evaluation device, the device comprising:

An acquisition module is used to acquire the text to be evaluated of the target object during the conversation;

An evaluation module, used for performing speech evaluation processing on the text to be evaluated through a text evaluation model to determine a text evaluation result of the text to be evaluated; wherein the text evaluation model is established by using a training method such as the text evaluation model;

An analysis module, used to determine whether the text to be evaluated needs to be optimized based on the text evaluation result;

The processing module is used to perform text optimization processing on the text to be evaluated to obtain an optimized text when text optimization is required.

In an eighth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

Get the text to be evaluated of the target object during the conversation;

Performing speech evaluation processing on the text to be evaluated by a text evaluation model to determine a text evaluation result of the text to be evaluated; wherein the text evaluation model is trained using the training method of the text evaluation model;

Based on the text evaluation result, determining whether the text to be evaluated needs to be optimized;

When text optimization is required, text optimization processing is performed on the text to be evaluated to obtain an optimized text.

In a ninth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

Get the text to be evaluated of the target object during the conversation;

Performing speech evaluation processing on the text to be evaluated by a text evaluation model to determine a text evaluation result of the text to be evaluated; wherein the text evaluation model is trained using the training method of the text evaluation model;

Based on the text evaluation result, determining whether the text to be evaluated needs to be optimized;

When text optimization is required, text optimization processing is performed on the text to be evaluated to obtain an optimized text.

In a tenth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

Get the text to be evaluated of the target object during the conversation;

Performing speech evaluation processing on the text to be evaluated by a text evaluation model to determine a text evaluation result of the text to be evaluated; wherein the text evaluation model is trained using the training method of the text evaluation model;

Based on the text evaluation result, determining whether the text to be evaluated needs to be optimized;

When text optimization is required, text optimization processing is performed on the text to be evaluated to obtain an optimized text.

The training method, text evaluation method and related device of the above-mentioned text evaluation model obtain the text to be evaluated of the target object during the conversation; perform speech evaluation processing on the text to be evaluated through the text evaluation model to determine the text evaluation result of the text to be evaluated; based on the text evaluation result, determine whether the text to be evaluated needs to be optimized; if text optimization is required, perform text optimization processing on the text to be evaluated to obtain the optimized text. Throughout the process, a text evaluation model that supports accurate text evaluation processing of the text to be evaluated is used to conduct a comprehensive evaluation of the text to be evaluated of the target object during the conversation, so as to accurately determine the text that needs to be optimized in the text to be evaluated, so as to perform accurate text optimization processing.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a diagram of an application environment of a text evaluation model training method and a text evaluation method in one embodiment;

FIG2 is a schematic diagram of a flow chart of a method for training a text evaluation model in one embodiment;

FIG3 is a schematic diagram of a flow chart of a text evaluation method in one embodiment;

FIG4 is a schematic diagram of a flow chart of a text evaluation method in another embodiment;

FIG5 is a schematic diagram of a process flow of a self-assessment process of a target object in a specific application example;

FIG6 is a schematic flow chart of a self-lifting process of a target object in a specific application example;

FIG7 is a block diagram of a training device for a text evaluation model in one embodiment;

FIG8 is a structural block diagram of a text evaluation device in one embodiment;

FIG. 9 is a diagram showing the internal structure of a computer device in one embodiment.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

Take the communication process between customer service agents and customers as an example. In the process of communication between customer service agents and customers, the most important thing is the communication skills of customer service agents, where skills refer to the skills of customer service agents in communicating with customers and solving user problems. In other words, customer service agents need to improve their communication skills as much as possible, so as to make the communication process with customers more convenient and reduce the number of customer complaints.

At present, if you want to improve the communication skills of customer service agents, you need to evaluate and process the communication texts of customer service agents. Specifically, the quality inspection center obtains the texts of the current customer service agents, and the quality inspection personnel of the quality inspection center perform certain analysis and processing on the texts to determine the texts to be optimized in the communication of the current customer service agents, and inform the current customer service agents of the texts to be optimized, so as to optimize the texts to a certain extent.

Furthermore, in the above scheme, the text is analyzed and processed by the quality inspectors in a certain manner, which is generally carried out by the quality inspectors in a multi-party quality inspection manner. That is, multiple quality inspectors conduct quality inspection on the text of the current customer service agent, and based on the quality inspection results of multiple quality inspectors, the comprehensive quality inspection results of the text of the current customer service agent are obtained, so as to determine the text to be optimized in the communication of the current customer service agent based on the comprehensive text quality inspection results.

However, the multi-party quality inspection method in the above method requires multiple parties to perform manual quality inspection operations. Once some quality inspectors make quality inspection errors due to various reasons, the final quality inspection results will also be greatly different. Therefore, the above method cannot accurately evaluate and process the text of the current customer service agent.

In this case, the present application proposes a training method for a text evaluation model, a text evaluation method and related devices. In the text evaluation model training process, first, determine the candidate simulation objects of the target object; the candidate simulation objects include the shadow simulation objects of the target object and the shadow simulation objects of the non-target object; determine the matching simulation objects that match the target object from the candidate simulation objects; establish a conversation between the target object and the matching simulation object, and obtain the conversation text of the matching simulation object and the text label of the conversation text during the conversation; based on the conversation text and the text label, train the text evaluation model. In the text evaluation method, first, obtain the text to be evaluated of the target object during the conversation; perform speech evaluation processing on the text to be evaluated through the text evaluation model to determine the text evaluation result of the text to be evaluated; based on the text evaluation result, determine whether the text to be evaluated needs to be optimized; if text optimization is required, perform text optimization processing on the text to be evaluated to obtain the optimized text.

In the whole process, the text evaluation model is trained by analyzing the conversation text and text labels obtained after the conversation process between the target object and the candidate simulation object. The trained text evaluation model can support accurate evaluation of the text to be evaluated. Furthermore, the candidate simulation object includes the shadow simulation object of the target object and the shadow simulation object of the non-target object. The model can be comprehensively trained through different training data, so that the multiple text evaluation models obtained through training can be used to perform comprehensive text evaluation processing on the text to be evaluated, thereby improving the accuracy of text evaluation. Furthermore, the model is trained by determining the shadow simulation object of the target object and the shadow simulation object of the non-target object, so that the target object can use the conversation method of the simulation object that is the same as or different from its own expression habits to achieve self-reflection and self-learning of the target object, improve the conversation level and evaluation ability of the target object, and then improve the accuracy of the text labels used for model training, thereby improving the accuracy of the text evaluation model.

The text evaluation method provided in the embodiment of the present application can be completed by a computer device, wherein the computer device can be a terminal or a server. The following will take the computer device as a terminal or a server as an example to describe in detail the execution process of the text evaluation method.

Taking the computer device as a server as an example, the text evaluation method provided in the embodiment of the present application can be applied to the application environment as shown in Figure 1, and the terminal 102 and the server 104 cooperate to perform the text evaluation task. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

In the training process of the text evaluation model, the processing personnel determines the candidate simulation objects of the target object in the model training interface of the terminal 102; the candidate simulation objects include the shadow simulation objects of the target object and the shadow simulation objects of the non-target object; the terminal 102 uploads the candidate simulation objects of the target object to the server 104, and the server 104 determines the candidate simulation objects of the target object; and based on the further selection operation of the processing personnel in the model training interface of the terminal 102, the terminal 102 determines the matching simulation objects that match the target object from the candidate simulation objects and forwards them to the server 104, and the server 104 determines the matching simulation objects that match the target object from the candidate simulation objects; the server 104 establishes a conversation between the target object and the matching simulation object, and obtains the conversation text of the matching simulation object and the text label of the conversation text during the conversation; based on the conversation text and the text label, the text evaluation model is trained. Further, the trained text evaluation model is stored in the database of the server 104.

Taking the computer device as terminal 102/server 104 as an example, the text evaluation method provided in the embodiment of the present application can also be applied to terminal 102/server 104 alone, and the terminal 102/server 104 independently performs the text evaluation operation. Specifically, the terminal 102/server 104 independently determines the candidate simulation object of the target object; the candidate simulation object includes the shadow simulation object of the target object and the shadow simulation object of the non-target object; the matching simulation object matching the target object is determined from the candidate simulation object; a conversation between the target object and the matching simulation object is established, and the conversation text of the matching simulation object and the text label of the conversation text are obtained during the conversation; based on the conversation text and the text label, the text evaluation model is trained. Further, the trained text evaluation model is stored in the corresponding database. During the text evaluation process, the processing personnel enters the conversation process number in the text processing interface of the terminal 102, and the terminal 102 generates a text processing request based on the conversation process number entered by the user, and uploads the text processing request to the server 104; the server 104 extracts the conversation process number in the text processing request, and queries the text to be evaluated of the target object corresponding to the conversation process number from the database. The server 104 performs speech evaluation processing on the text to be evaluated through different text evaluation models stored in the database to determine the text evaluation result of the text to be evaluated; based on the text evaluation result, it determines whether the text to be evaluated needs to be optimized; if text optimization is needed, the text to be evaluated is optimized to obtain the optimized text.

In other embodiments, the text evaluation method can also be applied to the terminal 102/server 104 alone. For example, the processing personnel enter the session process number in the text processing interface of the terminal 102, and the terminal 102 obtains the text to be evaluated of the target object in the session process corresponding to the session process number based on the session process number entered by the user; thereby, the above-mentioned text processing method is used to perform text processing on the text to be evaluated of the target object in the session process, which will not be described in detail here. For another example, the server 104 obtains the text to be evaluated of the target object in the session process from the database; and the above-mentioned text processing method is used to perform text processing on the text to be evaluated of the target object, which will not be described in detail here.

In some embodiments, as shown in FIG2 , a method for training a text evaluation model is provided, which may specifically include the following steps:

S100, determining candidate simulation objects of the target object.

Among them, the candidate simulation objects include the shadow simulation objects of the target object and the shadow simulation objects of the non-target object. In the present application, the target object is any manual seat, and the non-target object is the manual seat other than the target object. For example, there are manual seats (also called customer service, or manual customer service) A and manual seat B. If manual seat A is the target object, then manual seat B is the non-target object. If there are at least two manual seats other than the target object among the manual seats, the non-target object can be determined in the following manner: For example, each ungrouped manual seat can be scored to obtain a scoring result, and the scoring result includes a scoring grade and an actual score, and the scoring grades from good to bad are: Grade A, Grade B, and Grade C. The human agents are grouped based on the scoring levels. For example, all the human agents of level A are grouped into group A, all the human agents of level B are grouped into group B, and all the human agents of level C are grouped into group C. After the grouping is completed, any human agent in a group with a worse score than the group where the target object is located can be used as a non-target object, and any human agent in a group with a better score than the group where the target object is located can be used as a non-target object. The method of scoring the human agents includes but is not limited to scoring based on the previous work performance of the human agents.

For another example, the previous work performance of each human agent in the group where the target object is located can be scored. It should be noted that the grouping here refers to natural grouping, and is not grouped according to work performance. For example, it can be grouped according to the type of event handled, etc. After the previous work performance of the human agents in the group where the target object is located is scored according to the same scoring criteria, any human agent in the group where the target object is located that has a different scoring level from the target object and a large difference in scoring level or actual score is regarded as a non-target object.

The target object's shadow simulation object refers to a robot that has the same expression habits as the target object itself. It can be considered that the target object's shadow simulation object is a robot trained by learning the target object's words and expressions. Similarly, the non-target object's shadow simulation object refers to a robot that has the same expression habits as the non-target object itself. It can be considered that the non-target object's shadow simulation object is a robot trained by learning the non-target object's words and expressions.

Specifically, the shadow simulation object of each artificial seat is determined. Since the artificial seats include target objects and non-target objects, the shadow simulation object of the target object and the shadow simulation object of the non-target object are determined, and the shadow simulation object of the target object and the shadow simulation object of the non-target object are determined as candidate simulation objects of the target object.

S200: Determine a matching simulation object that matches the target object from candidate simulation objects.

Specifically, since the candidate simulation objects include the shadow simulation objects of the target object and the shadow simulation objects of the non-target object, the candidate simulation objects can be screened to determine the matching simulation objects that need to be matched with the target object, that is, to determine the matching simulation objects that need to have a conversation with the target object. The text evaluation model can be trained through the conversation between the target object and the matching simulation objects.

Furthermore, the method for screening candidate simulation objects includes but is not limited to: when the candidate simulation objects include a shadow simulation object of the target object, directly using the shadow simulation object of the target object as a matching simulation object to match the target object; when the candidate simulation objects include a shadow simulation object of a non-target object, determining the non-target object among the non-target objects that has the largest difference in actual score from the target object, and using the non-target object that has the largest difference from the target object as the matching simulation object.

S300, establishing a conversation between the target object and the matching simulation object, and obtaining the conversation text of the matching simulation object and the text label of the conversation text during the conversation.

Specifically, a conversation is established between the target object and the matching simulation object to identify the conversation text of the matching simulation object, and the conversation text of the matching simulation object and the text label of the conversation text are obtained during the conversation. Specifically, a conversation is established between the target object and the shadow simulation object of the target object, and the conversation text of the shadow simulation object of the target object and the text label of the conversation text of the shadow simulation object of the target object are obtained during the conversation; a conversation is established between the target object and the shadow simulation object of the non-target object, and the conversation text of the shadow simulation object of the non-target object and the text label of the conversation text of the shadow simulation object of the non-target object are obtained during the conversation.

Furthermore, obtaining the conversation text and text labels of the conversation text of the matching simulation object during the conversation includes: obtaining the conversation text during the conversation, removing the conversation text of the target object in the conversation text, and retaining the conversation text of the matching simulation object; pushing the conversation text of the matching simulation object to the terminal of the target object, and displaying it to the target object by the terminal of the target object; the target object evaluates the conversation text of the matching simulation object on the display interface, and determines the text label of the conversation text, wherein the text label includes text to be optimized and text that does not need to be optimized; associating the text label with the conversation text on the display interface, and uploading it to the terminal, and the terminal displays the text label of the conversation text to the server; the server obtains the text label of the conversation text. In addition, the target object can also upload the marking reason of the text label of the conversation text together with the text label of the conversation text to the server, and store the marking reason of the text label of the conversation text for retrospective analysis.

S400, training a text evaluation model based on the conversation text and the text label.

Specifically, the server trains the text evaluation model through the conversation text and the text label of the conversation text to obtain a trained text evaluation model. The text evaluation model supports evaluating the text when the text is input and outputting the text label of the text, wherein the text label includes the label of the text to be optimized and the label of the text that does not need to be optimized. In addition, the target object can also mark the conversation text of the target object itself during the conversation, marking the text that expresses dissatisfaction with the matching simulation object.

Furthermore, since the text label of the text evaluation model is marked by the target object, when the text evaluation model is used to process the text of the target object, it can be considered that the text evaluation model at this time is a self-evaluation model that can perform self-evaluation on the text of the target object.

In this embodiment, the conversation text and text labels obtained by analyzing the conversation process between the target object and the candidate simulation object are used to train the text evaluation model, and the text evaluation model obtained by training can support accurate evaluation of the text to be evaluated; wherein, the candidate simulation object includes the shadow simulation object of the target object and the shadow simulation object of the non-target object, and the comprehensive training model can be used through different training data, so that the subsequent multiple text evaluation models obtained by training can be used to perform comprehensive text evaluation processing on the text to be evaluated, thereby improving the accuracy of text evaluation. Further, the model is trained by determining the shadow simulation object of the target object and the shadow simulation object of the non-target object, so that the target object can use the conversation method of the simulation object that is the same as or different from its own expression habits to achieve self-reflection and self-learning of the target object, improve the conversation level and evaluation ability of the target object, and then improve the accuracy of the text label, thereby improving the accuracy of the text evaluation model.

In some embodiments, determining a candidate simulation object for the target object includes:

Obtain a preset initial text simulation model, determine a non-target object, and obtain the first historical session data of the target object and the second historical session data of the non-target object; based on the first historical session data, train the preset initial text simulation model belonging to the service role to obtain a shadow simulation object of the target object; based on the second historical session data, train the preset initial text simulation model to obtain a shadow simulation object of the non-target object.

Among them, the non-target object and the target object are different users belonging to service roles in the same business scenario, that is, manual agents; the shadow simulation object of the target object has the same individual characteristics as the target object, and having the same individual characteristics means that the shadow simulation object of the target object imitates the first historical conversation data to conduct a conversation during the conversation; the shadow simulation object of the non-target object has the same individual characteristics as the non-target object, and having the same individual characteristics means that the shadow simulation object of the non-target object imitates the second historical conversation data to conduct a conversation during the conversation.

Specifically, the historical conversation records of all human agents in a certain business scenario are obtained, and the historical conversation data of each human agent is input into the preset initial text simulation model. The preset initial text simulation model is a large language model. The large model refers to the large model with billions of parameters produced by knowledge extraction and learning through billions of corpora or images, and a large amount of unlabeled and very cheap data is used as a pre-trained model. The large language model will learn the words and expressions in the historical conversation records of each human agent, so as to simulate and generate a simulation robot with the characteristics of each human agent.

That is to say, each artificial agent will generate its own corresponding simulation robot, and the simulation robot corresponding to each artificial agent will imitate the historical conversation data of the artificial agent to conduct the conversation during the conversation.

Based on the first historical conversation data of the target object, the preset initial text simulation model is trained to obtain the shadow simulation object of the target object; based on the second historical conversation data of the non-target object, the preset initial text simulation model is trained to obtain the shadow simulation object of the non-target object. The shadow simulation object of the target object can imitate the first historical conversation data to conduct conversation in the subsequent conversation process; the shadow simulation object of the non-target object can imitate the second historical conversation data to conduct conversation in the subsequent conversation process.

In this embodiment, based on the historical conversation data of the artificial agent, a large language model is used to simulate the shadow simulation object of the artificial agent, so that the simulated shadow simulation object is a shadow simulation object with the same individual characteristics as the artificial agent itself, that is, the shadow simulation object will simulate the historical conversation method of the artificial agent in subsequent conversations, thereby improving the efficiency of obtaining candidate simulation objects, and further improving the efficiency of training the text evaluation model.

In some embodiments, establishing a session between a target object and a matching simulation object includes:

Assign a customer role to the target object and an agent role to the matching simulation object; establish a conversation between the customer role and the agent role.

Specifically, a customer role is assigned to the target object, and an agent role is assigned to the matching simulation object, and a conversation between the target object and the matching simulation object is used to simulate a conversation between the customer and the human agent. That is, when the matching simulation object is a shadow simulation object of the target object, the target object is used as the customer role, the matching simulation object is used as the agent role, and a conversation between the target object and the shadow simulation object of the target object is used to simulate a conversation between the customer and the human agent; when the matching simulation object is a shadow simulation object of a non-target object, the target object is used as the customer role, the matching simulation object is used as the agent role, and a conversation between the non-target object and the shadow simulation object of the non-target object is used to simulate a conversation between the customer and the human agent.

In this embodiment, by matching the matching simulation object with the target object, assigning a customer role to the target object, and assigning an agent role to the matching simulation object, it is possible to simulate the conversation between the customer and the agent through the conversation between the matching simulation object and the target object, thereby efficiently obtaining the conversation text of the matching simulation object and the text label of the conversation text during the conversation.

In some embodiments, the shadow simulation objects of the non-target objects include shadow simulation objects of first-category objects and shadow simulation objects of second-category objects. The first-category objects and the target objects are of the same type, and the second-category objects and the target objects are of different types.

The matching simulation objects include one or more of the following: a shadow simulation object of the target object, a shadow simulation object of the first type of object, and a shadow simulation object of the second type of object.

A matching simulation object corresponds to a text evaluation model, and the text evaluation model includes one or more of the following: when the matching simulation object includes a shadow simulation object of a target object, a first model is obtained based on the conversation text between the target object and the shadow simulation object of the target object; when the matching simulation object includes a shadow simulation object of a first category of objects, a second model is obtained based on the conversation text between the target object and the shadow simulation object of the first category of objects; when the matching simulation object includes a shadow simulation object of a second category of objects, a third model is obtained based on the conversation text between the target object and the shadow simulation object of the second category of objects.

Specifically, the first category of objects and the target object are of the same type, and the second category of objects and the target object are of different types. For example, target object A belongs to group 1. Since the requirements in the same group are more similar, the same or different types of objects of the target object can be marked according to the performance score or grade in the group. For example, objects in group 1 with the same performance grade as A can be regarded as objects of the same type as the target object, that is, objects of the first category; objects in group 1 with a large difference in performance grade from A can be regarded as objects of different types from the target object, that is, objects of the second category.

The shadow simulation objects of the first category of objects refer to the shadow simulation objects of the same category of the target object, and the shadow simulation objects of the second category of objects refer to the shadow simulation objects of objects of different categories of the target object. In other words, the shadow simulation objects of the first category of objects are shadow simulation objects that imitate the historical session data of objects of the same category of the target object, and the shadow simulation objects of the second category of objects are shadow simulation objects that imitate the historical session data of objects of different categories of the target object.

In the case where the shadow simulation object of the non-target object includes the shadow simulation object of the first category of object and the shadow simulation object of the second category of object, that is, the candidate simulation object includes the shadow simulation object of the target object, the shadow simulation object of the first category of object, and the shadow simulation object of the second category of object, the matching simulation object includes at least one of the shadow simulation object of the target object, the shadow simulation object of the first category of object, and the shadow simulation object of the second category of object.

Further, one matching simulation object corresponds to one text evaluation model. When the matching simulation object includes the shadow simulation object of the target object, the first model is trained based on the conversation text between the target object and the shadow simulation object of the target object, and the text label of the conversation text; when the matching simulation object includes the shadow simulation object of the first type of object, the second model is trained based on the conversation text between the target object and the shadow simulation object of the first type of object, and the text label of the conversation text; when the matching simulation object includes the shadow simulation object of the second type of object, the third model is trained based on the conversation text between the target object and the shadow simulation object of the second type of object, based on the conversation text between the target object and the shadow simulation object of the second type of object, and the text label of the conversation text. The trained model supports outputting the text evaluation result of the conversation text when the conversation text is input.

In this embodiment, by training multiple text evaluation models, and the text evaluation model is a model obtained after the target object evaluates the text of different types of simulation objects, therefore, multiple text evaluation models can be used to perform comprehensive text evaluation processing on the text to be evaluated; further, when performing text evaluation on the text to be evaluated of the target object, the text evaluation model is also a self-evaluation model, so that self-reflection can be achieved during the text evaluation process of the target object.

In some embodiments, as shown in FIG3 , a text evaluation method is provided, which may specifically include the following steps:

S500, obtaining the text to be evaluated of the target object during the conversation.

The two objects included in the conversation process are the target object and the customer object. In this application, the target object includes but is not limited to a human agent. When the target object is a human agent, the customer object is the object that has a conversation with the target object, such as the caller object that has a conversation with the human agent, or the outgoing caller object that has a conversation with the human agent. The text to be evaluated can be all the text of the target object during the conversation, or part of the text of all the text of the target object during the conversation, which can be customized through the settings of the processing personnel.

Specifically, the processing personnel inputs the session process number in the text processing interface of the terminal. The terminal generates a text processing request based on the session process number input by the user and uploads the text processing request to the server. The server extracts the session process number in the text processing request and searches the database for the text to be evaluated of the target object corresponding to the session process number. In addition, the text of the customer object corresponding to the session process number can also be obtained.

S600, performing speech evaluation processing on the text to be evaluated by using a text evaluation model to determine a text evaluation result of the text to be evaluated.

The text evaluation model includes one or more of the following:

In the case where the matching simulation object includes the shadow simulation object of the target object, the first model is obtained based on the conversation text between the target object and the shadow simulation object of the target object; in the case where the matching simulation object includes the shadow simulation object of the first type of object, the second model is obtained based on the conversation text between the target object and the shadow simulation object of the first type of object; in the case where the matching simulation object includes the shadow simulation object of the second type of object, the third model is obtained based on the conversation text between the target object and the shadow simulation object of the second type of object. The first model, the second model, and the third model all support outputting a text evaluation result of the text when the text is input.

The text to be evaluated is processed by the text evaluation model to determine the text evaluation result of the text to be evaluated. It is understandable that each text evaluation model corresponds to a text evaluation result of the text to be evaluated. The text evaluation model in the present application includes one or more models of the first model, the second model, and the third model, that is, the text evaluation result may also include one or more of the following:

When the text evaluation model includes a first model, the text evaluation result output by the first model; when the text evaluation model includes a second model, the text evaluation result output by the second model; when the text evaluation model includes a third model, the text evaluation result among the text evaluation results output by the third model.

The text evaluation model obtained through training processes the text to be evaluated to determine the text evaluation result of the text to be evaluated.

S700: Determine whether the text to be evaluated needs to be optimized based on the text evaluation result.

Specifically, since the text evaluation result is obtained by one or more text evaluation models, and one text evaluation model corresponds to one text evaluation result, when there are multiple text evaluation results, it is necessary to perform comprehensive processing on the text evaluation results, that is, to perform comprehensive processing on at least two obtained text evaluation results to obtain a comprehensive text evaluation result; further, based on the comprehensive text evaluation result, it is determined whether the text to be evaluated needs to be optimized. When there is only one text evaluation result, it is directly determined based on the text evaluation result whether the text to be evaluated needs to be optimized.

S800: When text optimization is required, text optimization processing is performed on the text to be evaluated to obtain an optimized text.

Specifically, if the text to be evaluated in this evaluation needs to be optimized, the text to be evaluated that needs to be optimized can be optimized to obtain the optimized text corresponding to the text to be evaluated. For example, the process of optimizing the text to be evaluated that needs to be optimized can be completed by deep learning, such as training a text optimization model, and performing text optimization on the text to be evaluated that needs to be optimized through the trained text optimization model.

In the above text processing method, the text to be evaluated of the target object during the conversation is obtained; the text to be evaluated is evaluated by the text evaluation model to determine the text evaluation result of the text to be evaluated; based on the text evaluation result, it is determined whether the text to be evaluated needs to be optimized; if text optimization is required, the text to be evaluated is optimized to obtain the optimized text. Throughout the whole process, a text evaluation model that supports accurate text evaluation of the text to be evaluated is used to comprehensively evaluate the text to be evaluated of the target object during the conversation, so that the text that needs to be optimized in the text to be evaluated can be accurately determined for accurate text optimization.

In some embodiments, the text evaluation model includes one or more of the following: a first model trained based on a conversation text between a target object and a shadow simulation object of the target object, a second model trained based on a conversation text between the target object and a shadow simulation object of a first type of object, and a third model trained based on a conversation text between the target object and a shadow simulation object of a second type of object;

One text evaluation model corresponds to one text evaluation result. Based on the text evaluation result, it is determined whether the text to be evaluated needs to be optimized, including:

Obtain the corresponding weight coefficient of each text evaluation result; obtain the comprehensive text evaluation result based on the weight coefficient and the text evaluation result; and determine whether the text to be evaluated needs to be optimized based on the comprehensive text evaluation result.

Specifically, a text evaluation model corresponds to a text evaluation result, and a text evaluation result corresponds to a weight coefficient. That is to say, different text evaluation results have different weight coefficients. The comprehensive text evaluation result is obtained based on the sum of the products of different text evaluation results and the weight coefficients corresponding to the text evaluation results.

For example, the texts to be evaluated are a and b. After the first model, the second model, and the third model process the texts to be evaluated, the ratio of the weight coefficients of the text evaluation results obtained is 7:2:1. If the text to be evaluated is a, the first model, the second model, and the third model are used to perform text evaluation on the text a to be evaluated, and the text evaluation result output by the first model is that text optimization is required, and the text evaluation results output by the second model and the third model are both that no text optimization is required, then based on each text evaluation result and its corresponding weight coefficient, the text comprehensive evaluation result is: 7. In other words, through the corresponding weight coefficient of each text evaluation result, the text evaluation result obtained by the third model is taken as 1 copy, the text evaluation result obtained by the second model is taken as 2 copies, and the text evaluation result obtained by the first model is taken as 7 copies. If there is a model whose text evaluation result is that text optimization is required, the number of copies corresponding to the model is added to the text comprehensive evaluation result. If the text evaluation result of the model is that text optimization is not required, there is no need to add or subtract the number of copies corresponding to the model from the text comprehensive evaluation result.

Continuing with the example of the text b to be evaluated, the first model, the second model, and the third model are used to perform text evaluation on the text b. The text evaluation result output by the first model is that no text optimization is required, and the text evaluation results output by the second model and the third model both indicate that text optimization is required. At this time, based on the text evaluation result and its corresponding weight coefficient, the comprehensive text evaluation result is: 3.

Compare the text comprehensive evaluation results of each text to be evaluated, and determine the text to be evaluated that needs to be optimized in this application. Further, in determining the text to be evaluated that needs to be optimized in this application, the text to be evaluated with the highest text comprehensive evaluation result can be used as the text to be evaluated that needs to be optimized, for example, the text to be evaluated a between the text to be evaluated a and the text to be evaluated b is used as the text to be evaluated that needs to be optimized; a preset optimization threshold can also be set, and the text to be evaluated that is greater than the preset optimization threshold in the text comprehensive evaluation result is used as the text to be evaluated that needs to be optimized, for example, if the optimization threshold is set to 2, then the text to be evaluated a and the text to be evaluated b are both texts to be evaluated that need to be optimized.

In this embodiment, by obtaining the corresponding weight coefficient of each text evaluation result, the comprehensive text evaluation result can be efficiently obtained based on the weight coefficient, and then it can be accurately determined whether the text to be evaluated needs text optimization, so as to optimize the text more efficiently.

In some embodiments, as shown in FIG. 4 , S800 includes:

S820: When text optimization is required, obtain second conversation texts generated by different conversations within a preset time period.

S840: Screening out a first candidate text in the second conversation text, the first candidate text having a similarity with the text to be evaluated greater than a preset similarity threshold.

S860: Determine an optimized text of the text to be evaluated based on the first candidate text.

The second conversation text includes the text to be evaluated of the target object, while the first candidate text does not include the text to be evaluated.

Specifically, the preset time period can be a certain time period in history, that is, the second conversation texts in different conversation processes corresponding to each manual seat in the preset time period can be obtained. The second conversation text is essentially the conversation text generated when each manual seat and the dialogue object of the manual seat have a conversation. Therefore, the second conversation text includes both the text of the manual seat and the text of the dialogue object with the manual seat. However, the present application needs to use the text of the manual seat in different conversation processes. Therefore, it is necessary to filter the text of the dialogue object with the manual seat in different conversation processes and retain the text of the manual seat in different conversation processes. The second conversation text also includes the text to be evaluated corresponding to the target object. It can be considered that the preset time period includes the time point when the text to be evaluated is generated. In addition, in actual applications, the third conversation texts in different conversation processes in a certain time period in history can also be obtained. The preset time period does not include the time point when the text to be evaluated is generated, and the text to be evaluated of the target object is obtained, and the text to be evaluated is combined with the third conversation text as the second conversation text.

The first candidate text whose similarity to the text to be evaluated is greater than the preset similarity threshold is screened from the second conversation text. In practical applications, the similarity threshold is generally set to 0.8. Only when the similarity to the text to be evaluated is greater than the preset similarity threshold in the second conversation text, the first candidate text obtained may be the optimized text of the text to be evaluated. It is understandable that the optimized text of the text to be evaluated is greater than the preset similarity threshold, but the text whose similarity to the text to be evaluated is greater than the preset similarity threshold is not necessarily the optimized text of the text to be evaluated, that is to say, the first candidate text is not necessarily the optimized text of the text to be evaluated, and it is also necessary to perform certain screening process on the first candidate text again, so that the text after the screening process can be used as the optimized text corresponding to the text to be evaluated. In addition, if the similarity to the text to be evaluated in the second conversation text is less than the preset similarity threshold, the difference between this part of text and the text to be evaluated is too large, and can not be used as the optimized text of the text to be evaluated.

In this embodiment, by obtaining the conversation texts of other human agents during the conversation within a preset time period, and by screening the first candidate text therefrom, since the first candidate text is a text whose similarity with the text to be evaluated is greater than a preset similarity threshold, therefore, by using the first candidate text as the optimized text corresponding to the text to be evaluated, it is possible to learn the excellent speech techniques of other human agents in similar situations encountered during the conversation, thereby achieving accurate optimization of the text to be evaluated.

In some embodiments, in the second conversation text, screening out a first candidate text having a similarity with the text to be evaluated greater than a preset similarity threshold, further comprising:

The second conversation text is clustered to obtain a clustering result; based on the clustering result, a second candidate text in the same cluster as the text to be evaluated is obtained; the similarity between the second candidate text and the text to be evaluated is determined, and a first candidate text whose similarity with the text to be evaluated is greater than a preset similarity threshold is screened from the second candidate text.

Specifically, the reason why the second conversation text includes the text to be evaluated of the target object is that the second conversation text needs to be clustered to determine the text in the second conversation text that is in the same cluster as the text to be evaluated. That is to say, the process of determining the first candidate text in the present application has undergone two screenings. Among them, the first screening is to obtain the second candidate text in the same cluster as the text to be evaluated; the second screening refers to screening the second candidate text to obtain the first candidate text whose similarity with the text to be evaluated is greater than a preset similarity threshold.

Furthermore, the clustering means include but are not limited to k-means clustering. The clustering process needs to specify the number of clusters, such as being able to specify how many categories the second conversation text is clustered into. The second conversation text is clustered to obtain a clustering result, which includes the final situation of the clustering, such as which texts are in the same cluster. A second candidate text in the same cluster as the text to be evaluated is obtained from the second conversation text, and the similarity between the second candidate text and the text to be evaluated is obtained.

When obtaining the similarity between the second candidate text and the text to be evaluated, it can be achieved through a pre-trained vector language model. Specifically, the pre-trained vector language model supports outputting the sentence vector corresponding to the text when the text is input. Among them, the sentence vector refers to the vector corresponding to a sentence of text. The second candidate text and the text to be evaluated are processed by the pre-trained vector language model to obtain the sentence vectors corresponding to the second candidate text and the text to be evaluated. Based on the sentence vector of the second candidate text and the sentence vector of the text to be evaluated, the similarity between the second candidate text and the text to be evaluated is determined.

Based on the sentence vector of the second candidate text and the sentence vector of the text to be evaluated, the method for determining the similarity between the second candidate text and the text to be evaluated includes: obtaining the similarity between each text in the second candidate text and the text to be evaluated by performing vector dot multiplication between the sentence vector of each text in the second candidate text and the sentence vector of the text to be evaluated, and performing secondary screening on the second candidate text based on the similarity to obtain the first candidate text whose similarity with the text to be evaluated is greater than a preset similarity threshold.

In this embodiment, when the first candidate text whose similarity with the text to be evaluated is greater than the preset similarity threshold is screened in the second conversation text, a two-screening process is essentially performed. First, the second candidate text in the same cluster as the text to be evaluated is obtained; second, the first candidate text whose similarity with the text to be evaluated is greater than the preset similarity threshold is screened in the second candidate text. Through the two-screening process, the amount of calculation during similarity calculation can be reduced, and each text in the second conversation text does not need to obtain the similarity with the text to be evaluated by multiplying the sentence vector, thereby improving the efficiency of optimized text screening, and further improving the efficiency of text processing.

In some embodiments, determining an optimized text of the text to be evaluated based on the first candidate text includes:

The first candidate text is evaluated by using N text evaluation models to determine N text evaluation results of the first candidate text; if the N text evaluation results of the first candidate text all indicate that no text optimization is required, an optimized text of the text to be evaluated is determined based on the first candidate text.

Specifically, the first candidate text whose similarity with the text to be evaluated is greater than the preset similarity threshold is not necessarily the optimized text corresponding to the text to be evaluated. It is understandable that the first candidate text may be the optimized text of the text to be evaluated, and the text to be evaluated may also be the optimized text of the first candidate text. In layman's terms, the first candidate text may be a text with worse rhetoric than the text to be evaluated, or it may be a text with better rhetoric than the text to be evaluated. It is necessary to screen out texts with better rhetoric than the text to be evaluated from the first candidate text.

Specific screening means may include: using N text evaluation models, each text evaluation model corresponds to a text evaluation result, determining N text evaluation results of the first candidate text, and the text evaluation result characterizes whether the first candidate text needs to be optimized. If the first candidate text does not need to be optimized, then the first candidate text at this time is not only a text whose similarity with the text to be evaluated is greater than a preset similarity threshold, but also a text with better rhetoric than the text to be evaluated. Therefore, the first candidate text that does not need to be optimized can be used as the optimized text of the text to be evaluated. If the first candidate text needs to be optimized, then the first candidate text at this time cannot be determined whether it is better than the rhetoric of the text to be evaluated, and cannot be used as the optimized text of the text to be evaluated.

Furthermore, the N text evaluation models adopted include: a first model, a second model, and a third model. When determining the text evaluation result corresponding to the first candidate text, one or more models among the first model, the second model, and the third model perform text evaluation on the first candidate text, and the text evaluation result corresponding to each model performing text evaluation needs to be that no text optimization is required.

In this embodiment, when a first candidate text is screened out whose similarity with the text to be evaluated is greater than a preset similarity threshold, N text evaluation models are also required to perform a certain text evaluation on the first candidate text, so that the first candidate text is a text with better wording than the text to be evaluated, to avoid the first candidate text being a text with worse wording than the text to be evaluated, and failing to optimize the text to be evaluated.

In some embodiments, the conversation objects of the conversation process include a target object and a client object having a conversation with the target object;

Determining an optimized text of the text to be evaluated based on the first candidate text includes:

Obtain the interactive text of the text to be evaluated during the conversation; if the interactive text belongs to the text to be appeased, determine the appeased text of the interactive text; and concatenate the appeased text with the first candidate text to obtain the optimized text of the text to be evaluated.

Among them, the interactive text is generated by the customer object during the conversation.

Specifically, the optimized text in this application includes not only the first candidate text, but also the reassuring text when the customer needs to be reassured. For example, the reassuring text can be: "Don't worry." The reassuring text is spliced with the first candidate text to obtain the optimized text of the text to be evaluated.

Furthermore, the means of obtaining the soothing text includes: obtaining the interactive text of the text to be evaluated during the conversation. It can be understood that there are two objects in the conversation process, including the target object and the customer object, and the target object and the customer object have a conversation. The text of the customer object that is talking to the manual agent during this conversation is obtained, and the text of the customer object is used as the interactive text of the text to be evaluated generated by the manual agent; it should be explained that the interactive text here means: the interactive text can be the previous sentence text of the text to be evaluated by the manual agent, or it can be the next sentence text of the text to be evaluated by the manual agent. And the interactive text corresponds one-to-one with the text to be evaluated.

The interactive text is judged whether it is a text to be appeased, and a judgment result of the interactive text is obtained. The judgment result includes whether it belongs to the text to be appeased or not. The judgment of whether the interactive text is a text to be appeased is realized by the emotional appeased model. The training process of the emotional appeased model includes:

A. Obtain training data. For example, the training data may include: a first training text: "What is my relationship with him? What does it have to do with you?", and the emotion label corresponding to the first training text: "Refuse to disclose the relationship, wait to be appeased"; a second training text: "Your interest is too high", and the emotion label corresponding to the second training text: "Questioning the interest, wait to be appeased"; a third training text: "How do I operate?", and the emotion label corresponding to the second training text: "Operation consultation, no need to be appeased".

B. Model training process. Load a pre-trained language model, such as the Roberta model, and obtain

The training data is input into the pre-trained language model, and the classification task is set to start training. The trained emotional comfort model is obtained.

The interactive text is processed by the trained emotional comfort model to determine whether the interactive text of the customer object is a text to be comforted. If so, it is necessary to further determine how to comfort the interactive text. If not, there is no need to comfort the interactive text.

In the process of further determining how to appease the interactive text, an emotional appeasement model can be used to output the appeasement text corresponding to the text to be appeased. It should be explained that the emotional appeasement model is different from the emotional to-be-appeased model. The emotional appeasement model is a large model, that is, the emotional appeasement model is a generative model that can directly output the appeasement text corresponding to the text to be appeased for the customer object. The emotional to-be-appeased model is a discriminative model that can classify whether the text of the customer object belongs to the text to be appeased.

The training process of the emotional soothing model includes:

A. Load a generative pre-trained language model, such as the T5 model.

B. Obtain training data. For example, the training data of the emotional comfort model includes: "The fourth training data: What is my relationship with him? What does it have to do with you?", and the comfort text corresponding to the fourth training data: "The lender has left your phone number, please rest assured that we will keep it confidential for you"; the fifth training data: "Your interest rate is too high", and the comfort text corresponding to the fifth training data: "The current interest rate is..., it is compliant, please rest assured".

C. Based on the training data, the loaded generative pre-trained language model is trained to obtain a trained emotion soothing model.

The interactive text belonging to the text to be soothed is processed through the trained emotion soothing model, and the soothing text corresponding to the text to be soothed is output.

In this embodiment, the optimized text of the text to be evaluated is obtained by splicing the soothing text and the first candidate text, so that the obtained optimized text is not only a semantic optimization of the original text to be evaluated, but also can soothe the customer object, making the optimized text more effective and accurate. Furthermore, the emotional soothing model is used to judge whether the interactive text of the customer object needs to be soothed, and the interactive text belonging to the text to be soothed is obtained, and the soothing text corresponding to the text to be soothed is determined by the emotional soothing model. The use of the model can make the judgment result and the output of the soothing text more efficient and accurate.

In some embodiments, when the text to be evaluated does not need to be optimized, it is still necessary to determine the interactive text of the customer object during the conversation, and the interactive text is the text corresponding to the text to be evaluated of the target object; if the interactive text belongs to the text to be appeased, the appeased text corresponding to the text to be appeased is determined; and the appeased text is used as the optimized text of the text to be evaluated. That is, the appeased text and the superior text are generated independently, wherein the superior text is a text whose similarity with the text to be evaluated is greater than a preset similarity threshold, and the results obtained by processing N text evaluation models are all texts that do not need to be optimized. There may be no appeased text in the optimized text, but there is an excellent text; there may be appeased text in the optimized text, but there is no excellent text; there may be both appeased text and excellent text in the optimized text; there may be no appeased text and excellent text in the optimized text.

In some embodiments, the text processing of the present application is essentially a self-assessment and self-improvement process of the target object. FIG5 shows the self-assessment process of the target object, and FIG6 shows the self-improvement process of the target object, which specifically includes the following steps:

1. The self-assessment process of the target object:

1. Suppose there are human agents A, B, C, D, E, and F in a group of a certain agent center; obtain all historical conversation texts of human agents A, B, C, D, E, and F.

2. Input the historical conversation texts corresponding to each artificial agent A, B, C, D, E, F into the language model, and simulate and generate robots a, b, c, d, e, f with the individual characteristics of each artificial agent A, B, C, D, E, F, that is, candidate simulation objects a, b, c, d, e, f. Artificial agent A is the target object for evaluating her text. Artificial agent B is an object with the same performance level as artificial agent A, and artificial agent F is an object with a very different performance level from artificial agent A. At this time, artificial agent B can be regarded as an object of the same type as artificial agent A, and artificial agent F can be regarded as an object of a different type from artificial agent A. On this basis, robot a is regarded as the shadow simulation object of artificial agent A, robot b is regarded as the shadow simulation object of the first type of object, and robot f is regarded as the shadow simulation object of the second type of object.

3. Assign human agent A the identity of a customer, and assign robots a, b, and f corresponding to human agents A, B, and F as the identities of agents. Human agent A will have a conversation with the robot agent, that is, human agent A will imitate the way a customer calls and have a conversation with each robot agent a, b, and f respectively.

4. When human agent A and robot agent a are talking, the conversation text between human agent A and robot agent a is obtained, and the conversation text is pushed to human agent A. Human agent A marks the text labels that need to be optimized and the text labels that do not need to be optimized in the text of robot agent a, and marks the text that expresses dissatisfaction with the text of the agent. The text and text labels of robot agent a are used as training data to train the text evaluation model to obtain the first model that has been trained. The first model supports outputting the text evaluation label corresponding to the input text.

5. When human agent A and robot agent B are talking, the conversation text between human agent A and robot agent B is obtained, and the conversation text is pushed to human agent A. Human agent A marks the text labels that need to be optimized and the text labels that do not need to be optimized in the text of robot agent B, and marks the text that expresses dissatisfaction with the text of the agent. The text and text labels of robot agent B are used as training data to train the text evaluation model to obtain a trained second model. The second model supports outputting the text evaluation label corresponding to the input text.

6. When human agent A and robot agent f are talking, the conversation text between human agent A and robot agent f is obtained, and the conversation text is pushed to human agent A. Human agent A marks the text labels that need to be optimized and the text labels that do not need to be optimized in the text of robot agent f, and marks the text that expresses dissatisfaction with the text of the agent. The text and text labels of robot agent f are used as training data to train the text evaluation model, and a third model that has been trained is obtained. The third model supports outputting the text evaluation label corresponding to the text when the text is input.

2. The self-improvement process of the target object:

1. Obtain text 1 to be evaluated, text 2 to be evaluated, and text 3 to be evaluated of agent A during the current conversation; obtain previous sentence text 4 of the customer corresponding to text 1 to be evaluated, previous sentence text 5 of the customer corresponding to text 1 to be evaluated, and previous sentence text 6 of the customer corresponding to text 1 to be evaluated during the current conversation.

2, text to be evaluated 1, text to be evaluated 2 and text to be evaluated 3 are inputted into the first model, the second model and the third model respectively. Taking text to be evaluated 1 being inputted into the first model, the second model and the third model respectively as an example, each model outputs whether text to be evaluated 1 is the text that needs to be optimized for text, and based on the output result of each model, determines whether final text to be evaluated 1 is the text that needs to be optimized for text. Adopt the same step to process text to be evaluated 2 and text to be evaluated 3 respectively, and compare the text evaluation results of three texts to be evaluated, obtain the text to be evaluated that most urgently needs text optimization among the three texts to be evaluated. Here, text to be evaluated 1 is used as the text to be evaluated that most urgently needs text optimization.

3. Input the previous sentence text 4 of the customer corresponding to the evaluation text 1 into the emotional soothing model. The emotional soothing model supports outputting whether the text represents the result of needing soothing when the text is input. At this time, let the emotional soothing model output text 4 as the text to be soothed, and then input text 4 into the emotional soothing model. The emotional soothing model supports outputting the soothing text corresponding to the text when the text to be soothed is input. At this time, the emotional soothing model outputs the soothing text corresponding to text 4.

4. Determine the superior text of the text to be evaluated 1. Among them, the superior text of the text to be evaluated 1 needs to have a similarity with the text to be evaluated higher than 0.8, and is a text that does not need to be optimized. That is to say, when the superior text is input into the first model, the second model, and the third model, the text evaluation results output by the three models should all be text optimization is not required. Further, the judgment step of the similarity with the text to be evaluated higher than 0.8 is: first, obtain the historical conversation text data of all artificial seats A, B, C, D, E, and F, and use clustering to determine the texts in the historical conversation text data that are in the same cluster as the text to be evaluated, and calculate the similarity with the text to be evaluated respectively for the text in the same cluster as the text to be evaluated, and determine the texts with a similarity higher than 0.8 with the text to be evaluated.

5. Splice the comfort text and the alternative text to obtain the optimized text of the text to be evaluated 1.

In addition, if the text to be evaluated is a text that does not require text optimization, the previous sentence text of the customer corresponding to the text to be evaluated is input into the emotional soothing model. If the emotional soothing model outputs that the previous sentence text of the customer is the text to be soothed, the previous sentence text of the customer is input into the emotional soothing model, and the emotional soothing model outputs the soothing text corresponding to the previous sentence text of the customer. The soothing text is used as the optimized text.

If the text to be evaluated is text that does not require text optimization, and the emotional comfort model outputs that the customer's previous sentence text does not require comfort, then there is no need to optimize the text to be evaluated.

It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application also provides a text evaluation model training device for implementing the above-mentioned text evaluation model training method, and a text evaluation device for implementing the above-mentioned text evaluation method. The implementation scheme for solving the problem provided by the above-mentioned device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of the training device for one or more text evaluation models provided below can refer to the limitations of the training method for the text evaluation model above, and the specific limitations in the embodiments of one or more text evaluation devices can refer to the limitations of the text evaluation method above, which will not be repeated here.

In some embodiments, as shown in FIG. 7 , a training device for a text evaluation model is provided, comprising: a determination module 100 , a matching module 200 , a conversation module 300 , and a training module 400 , wherein:

A determination module 100 is used to determine candidate simulation objects of a target object; the candidate simulation objects include shadow simulation objects of the target object and shadow simulation objects of non-target objects;

A matching module 200, for determining a matching simulation object that matches the target object from the candidate simulation objects;

The conversation module 300 is used to establish a conversation between the target object and the matching simulation object, and obtain the conversation text of the matching simulation object and the text label of the conversation text during the conversation;

The training module 400 is used to train the text evaluation model based on the conversation text and the text label.

In some embodiments, the determination module 100 is also used to obtain a preset initial text simulation model, determine a non-target object, and obtain first historical session data of the target object and second historical session data of the non-target object; the non-target object and the target object are different users belonging to a service role in the same business scenario; based on the first historical session data, the preset initial text simulation model is trained to obtain a shadow simulation object of the target object; the shadow simulation object of the target object has the same individual characteristics as the target object, and having the same individual characteristics means that the shadow simulation object of the target object imitates the first historical session data to conduct a conversation during the conversation; based on the second historical session data, the preset initial text simulation model is trained to obtain a shadow simulation object of the non-target object; the shadow simulation object of the non-target object has the same individual characteristics as the non-target object, and having the same individual characteristics means that the shadow simulation object of the non-target object imitates the second historical session data to conduct a conversation during the conversation.

In some embodiments, the conversation module 300 is further used to assign a customer role to the target object and an agent role to the matching simulation object; and to establish a conversation between the customer role and the agent role.

In some embodiments, the shadow simulation objects of the non-target objects include shadow simulation objects of first-category objects and shadow simulation objects of second-category objects, the first-category objects and the target objects are of the same type, and the second-category objects and the target objects are of different types;

Matching simulation objects include one or more of the following:

The shadow simulation object of the target object, the shadow simulation object of the first category of objects and the shadow simulation object of the second category of objects; one matching simulation object corresponds to a text evaluation model, and the text evaluation model includes one or more of the following: when the matching simulation object includes the shadow simulation object of the target object, a first model is obtained based on the conversation text between the target object and the shadow simulation object of the target object; when the matching simulation object includes the shadow simulation object of the first category of objects, a second model is obtained based on the conversation text between the target object and the shadow simulation object of the first category of objects; when the matching simulation object includes the shadow simulation object of the second category of objects, a third model is obtained based on the conversation text between the target object and the shadow simulation object of the second category of objects.

In some embodiments, as shown in FIG8 , a text evaluation device is provided, including: an acquisition module 500, an evaluation module 600, an analysis module 700, and a processing module 800, wherein:

An acquisition module 500 is used to acquire the text to be evaluated of the target object during the conversation;

An evaluation module 600 is used to perform a speech evaluation process on the text to be evaluated by using a text evaluation model to determine a text evaluation result of the text to be evaluated; wherein the text evaluation model is established by using a training method of a text evaluation model; an analysis module 700 is used to determine whether the text to be evaluated needs to be optimized based on the text evaluation result;

The processing module 800 is used to perform text optimization processing on the text to be evaluated to obtain an optimized text when text optimization is required.

In some embodiments, the text evaluation model includes one or more of the following: a first model obtained based on the conversation text training between the target object and the shadow simulation object of the target object, a second model obtained based on the conversation text training between the target object and the shadow simulation object of the first category of objects, and a third model obtained based on the conversation text training between the target object and the shadow simulation object of the second category of objects; one text evaluation model corresponds to one text evaluation result; the analysis module 700 is also used to obtain the weight coefficient of each text evaluation result; based on the weight coefficient and the text evaluation result, obtain a comprehensive text evaluation result; based on the comprehensive text evaluation result, determine whether the text to be evaluated needs text optimization.

In some embodiments, the processing module 800 is also used to obtain a second conversation text generated by different conversations within a preset time period when text optimization is required, the second conversation text including the text to be evaluated of the target object; in the second conversation text, screen out a first candidate text whose similarity with the text to be evaluated is greater than a preset similarity threshold; the first candidate text does not include the text to be evaluated; and determine the optimized text of the text to be evaluated based on the first candidate text.

In some embodiments, the processing module 800 is also used to cluster the second conversation text to obtain a clustering result; based on the clustering result, obtain a second candidate text in the same cluster as the text to be evaluated; determine the similarity between the second candidate text and the text to be evaluated, and screen the first candidate text from the second candidate text whose similarity with the text to be evaluated is greater than a preset similarity threshold.

In some embodiments, the processing module 800 is also used to evaluate the first candidate text through N text evaluation models to determine N text evaluation results of the first candidate text; if the N text evaluation results of the first candidate text all indicate that no text optimization is required, then the optimized text of the text to be evaluated is determined based on the first candidate text.

In some embodiments, the conversation objects of the conversation process include a target object and a customer object that dialogues with the target object; the processing module 800 is also used to obtain the interactive text of the text to be evaluated during the conversation, and the interactive text is generated by the customer object during the conversation; if the interactive text belongs to the text to be appeased, the appeasing text of the interactive text is determined; the appeasing text and the first candidate text are spliced to obtain the optimized text of the text to be evaluated.

Each module in the above text evaluation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG9. The computer device includes a processor, a memory, an input/output interface (Input/Output, referred to as I/O) and a communication interface. The processor, the memory and the input/output interface are connected via a system bus, and the communication interface is connected to the system bus via the input/output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as the text to be evaluated of the target object during the session. The input/output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a text processing method is implemented.

Those skilled in the art will understand that the structure shown in FIG. 9 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

In some embodiments, a computer device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims (15)

1. A method of training a text evaluation model, the method comprising:

Determining candidate simulation objects of the target object; the candidate simulation objects comprise shadow simulation objects of the target object and shadow simulation objects of non-target objects;

Determining a matching simulation object matched with the target object from the candidate simulation objects;

Establishing a session between the target object and the matching simulation object, and acquiring a session text of the matching simulation object and a text label of the session text in the session process;

and training a text evaluation model based on the session text and the text labels.

2. The method of claim 1, wherein determining candidate simulated objects for the target object comprises:

acquiring a preset initial text simulation model, determining a non-target object, and acquiring first historical session data of the target object and second historical session data of the non-target object; the non-target object and the target object are different users belonging to a service role in the same service scene;

Training the preset initial text simulation model based on the first historical session data to obtain a shadow simulation object of the target object; the shadow simulation object of the target object has the same individual characteristic as the target object, wherein the same individual characteristic means that the shadow simulation object of the target object simulates the first historical session data to perform a session in the session process;

Training the preset initial text simulation model based on the second historical session data to obtain a shadow simulation object of the non-target object; the shadow simulation object of the non-target object has the same individual characteristic as the non-target object, wherein the same individual characteristic means that the shadow simulation object of the non-target object simulates the second historical session data to conduct a session in the session process.

3. The method of claim 1, wherein the establishing a session between the target object and the matching simulated object comprises:

assigning a client role to the target object and an agent role to the matching simulation object;

And establishing a session between the client role and the agent role.

4. The method of claim 1, wherein the shadow simulated objects of the non-target object comprise shadow simulated objects of a first class of objects and shadow simulated objects of a second class of objects, the first class of objects being homogeneous objects with the target object and the second class of objects being heterogeneous objects with the target object;

The matching simulation object includes one or more of the following:

The shadow simulation object of the target object, the shadow simulation object of the first class object and the shadow simulation object of the second class object;

A matching simulated object corresponds to a text evaluation model, the text evaluation model comprising one or more of:

In the case that the matching simulation object comprises a shadow simulation object of the target object, training a first model based on a session text between the target object and the shadow simulation object of the target object;

in the case that the matching simulation object comprises a shadow simulation object of the first class object, training a second model based on a session text between the target object and the shadow simulation object of the first class object;

And training a third model based on the session text between the target object and the shadow simulation object of the second class object under the condition that the matching simulation object comprises the shadow simulation object of the second class object.

5. A method of text evaluation, the method comprising:

acquiring a text to be evaluated of a target object in a conversation process;

performing speaking evaluation processing on the text to be evaluated through a text evaluation model, and determining a text evaluation result of the text to be evaluated; wherein the text evaluation model is trained using the method of any one of claims 1-4;

determining whether the text to be evaluated needs text optimization or not based on the text evaluation result;

And under the condition that text optimization is required, text optimization processing is carried out on the text to be evaluated, and optimized text is obtained.

6. The method of claim 5, wherein the text evaluation model comprises one or more of: a first model obtained based on session text training between a target object and a shadow simulation object of the target object, a second model obtained based on session text training between the target object and a shadow simulation object of the first class object, and a third model obtained based on session text training between the target object and a shadow simulation object of the second class object;

A text evaluation model corresponds to a text evaluation result; the determining whether the text to be evaluated needs text optimization based on the text evaluation result comprises the following steps:

acquiring a weight coefficient of each text evaluation result;

based on the weight coefficient and the text evaluation result, obtaining a text comprehensive evaluation result;

And determining whether the text to be evaluated needs text optimization or not based on the text comprehensive evaluation result.

7. The method according to claim 5, wherein, in the case where text optimization is required, performing text optimization processing on the text to be evaluated to obtain an optimized text, including:

Under the condition that text optimization is required, second session texts generated by different sessions in a preset time period are obtained, wherein the second session texts comprise texts to be evaluated of the target object;

Screening to obtain a first candidate text with the similarity with the text to be evaluated being greater than a preset similarity threshold value in the second session text, wherein the first candidate text does not comprise the text to be evaluated;

and determining optimized text of the text to be evaluated based on the first candidate text.

8. The method according to claim 7, wherein the screening the second text for the first candidate text having a similarity with the text to be evaluated greater than a preset similarity threshold includes:

clustering the second session text to obtain a clustering result;

acquiring a second candidate text under the same cluster with the text to be evaluated based on the clustering result;

And determining the similarity between the second candidate text and the text to be evaluated, and screening the first candidate text with the similarity greater than a preset similarity threshold value from the second candidate text.

9. The method of claim 7, wherein the determining the optimized text of the text under evaluation based on the first candidate text comprises:

evaluating the first candidate text through N text evaluation models, and determining N text evaluation results of the first candidate text;

And if all the N text evaluation results of the first candidate text indicate that text optimization is not needed, determining the optimized text of the text to be evaluated based on the first candidate text.

10. The method of claim 7, wherein the session object of the session procedure comprises the target object and a client object that dialogues with the target object;

the determining the optimized text of the text to be evaluated based on the first candidate text comprises the following steps:

acquiring an interactive text of the text to be evaluated in the conversation process, wherein the interactive text is generated by a client object in the conversation process;

if the interactive text belongs to the text to be pacified, determining a pacifying text of the interactive text;

and splicing the pacifying text and the first candidate text to obtain an optimized text of the text to be evaluated.

11. A training device for a text evaluation model, the device comprising:

The determining module is used for determining candidate simulation objects of the target object; the candidate simulation objects comprise shadow simulation objects of the target object and shadow simulation objects of non-target objects;

The matching module is used for determining a matching simulation object matched with the target object from the candidate simulation objects;

The conversation module is used for establishing a conversation between the target object and the matched simulation object and acquiring a conversation text of the matched simulation object and a text label of the conversation text in a conversation process;

And the training module is used for training the text evaluation model based on the session text and the text label.

12. A text evaluation apparatus, the apparatus comprising:

the acquisition module is used for acquiring the text to be evaluated of the target object in the conversation process;

the evaluation module is used for performing speaking evaluation processing on the text to be evaluated through a text evaluation model and determining a text evaluation result of the text to be evaluated; wherein the text evaluation model is built using the method of any one of claims 1-4;

The analysis module is used for determining whether the text to be evaluated needs text optimization or not based on the text evaluation result;

and the processing module is used for carrying out text optimization processing on the text to be evaluated under the condition that text optimization is required to be carried out, so as to obtain an optimized text.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 10 when the computer program is executed.

14. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 10.

15. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any one of claims 1 to 10.