CN115577285A - Training set processing method, device, electronic device and storage medium for classification - Google Patents

Abstract

The invention provides a training set processing method and device for classification, an electronic device and a storage medium, wherein the method comprises the following steps: obtaining a classification sample set; determining a respective prediction classification corresponding to each sample, a probability of each prediction classification, and a disambiguation score corresponding to each sample; for each sample, determining the most similar sample corresponding to the sample from the samples with the target prediction classification corresponding to the sample; the most similar prediction classification corresponding to each classification label is determined based on the disambiguation score, the first target sample of which the prediction classification is the most similar prediction classification is determined from all samples with the classification labels based on the most similar prediction classification, the second target sample of which the prediction classification is the classification label is determined from all samples with the most similar prediction classification, and the defects of confusion of the classes or labeling errors of the samples and the like can be rapidly identified by analyzing the first target sample, the second target sample and the most similar sample corresponding to each sample, so that the sample cleaning speed is improved.

Description

Training set processing method, device, electronic device and storage medium for classification

technical field

The present invention relates to the technical field of text classification, in particular, to a training set processing method, device, electronic equipment and storage medium for classification.

Background technique

Classification tasks play a big role in many scenarios, such as the intention recognition of intelligent customer service, the recognition of flowers, plants and animals in nature, etc. In addition to relying on classification model technology, classification tasks also rely heavily on classification corpus, and the classification effect is related to the data quality and samples of the training set. The number has a lot to do with it. When there are a large number of labeling errors in the labeled samples of the training set, the performance of the algorithm will be affected and the classification effect will be seriously affected.

Contents of the invention

One of the objectives of the present invention is to provide a training set processing method, device, electronic equipment and storage medium for classification, which can quickly perform data cleaning on the training set, and identify samples or categories with wrong labels and confusing labels. An example can be implemented like this:

In a first aspect, the present invention provides a training set processing method for classification, the method comprising:

Obtaining a classification sample set; the classification sample set includes a plurality of samples and classification labels corresponding to each of the samples; one type of classification label corresponds to at least one of the samples;

Determining each predicted classification corresponding to each of the samples, the probability of each of the predicted classifications, and the disambiguation score corresponding to each of the samples; wherein, the disambiguation score represents the classification label corresponding to the sample the degree of error;

For each sample, from each sample with the target prediction classification corresponding to the sample, determine the most similar sample corresponding to each of the samples, and the probability of the target prediction classification is the largest;

Based on the disambiguation score, determine the most similar prediction classification corresponding to each of the classification labels, and based on the most similar prediction classification, from all samples with the classification labels, determine the prediction classification as the most similar prediction classifying the first target sample, and from among all samples having the most similar predicted class, determining a second target sample predicted to be classified as the class label;

Wherein, the first target sample and the second target sample are used to indicate that a merging strategy is performed on the classification label and the most similar predicted classification, or indicate that the first target sample and/or the second Two target samples implement a correction strategy; the most similar sample is used to indicate to correct the sample.

In a second aspect, the present invention provides a training set processing device for classification, comprising:

An acquisition module, configured to acquire a classification sample set; the classification sample set includes a plurality of samples and classification labels corresponding to each of the samples; one type of classification label corresponds to at least one of the samples;

A determining module, configured to determine each predicted classification corresponding to each of the samples, the probability of each of the predicted classifications, and the disambiguation score corresponding to each of the samples; wherein the disambiguation score represents the corresponding the error score for the class label;

The determination module is further configured to, for each sample, determine the most similar sample corresponding to each sample from each sample with the target prediction classification corresponding to the sample, and the probability of the target prediction classification is the largest ; The probability of the target prediction classification is the largest;

The determining module is further configured to determine the most similar predicted classification corresponding to each of the classification labels based on the disambiguation score, and determine from all samples with the classification labels based on the most similar predicted classification predicting a first target sample that is classified as the most similar predicted class, and determining a second target sample that is predicted to be the class label from all samples with the most similar predicted class;

Wherein, the first target sample and the second target sample are used to indicate that a merging strategy is performed on the classification label and the most similar predicted classification, or indicate that the first target sample and/or the second Two target samples implement a correction strategy; the most similar sample is used to indicate to correct the sample.

In a third aspect, the present invention provides an electronic device, including a processor and a memory, the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the computer program provided in the first aspect. A training set processing method for classification.

In a fourth aspect, the present invention provides a readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the training set processing method for classification as provided in the first aspect is implemented.

The present invention provides a training set processing method, device, electronic equipment and storage medium for classification. The method includes: first obtaining a classification sample set, and then predicting the category of each sample and the probability of each category. Since each sample is With a correct classification label, then for each category, from the prediction results of the samples with this category, it is possible to determine the predicted category with the largest conflict with the category, and the first target sample corresponding to the predicted category, And from each sample with the predicted classification, determine the second target sample whose prediction result is the category, and determine whether the categories need to be merged or whether the samples need to be adjusted by analyzing the first target sample and the second target sample , can also determine the most similar sample corresponding to each sample, and then determine whether the category of the sample needs to be adjusted. The whole process can quickly identify labeling errors and labeling confusion, and improve the speed of sample cleaning.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and thus It should be regarded as a limitation on the scope, and those skilled in the art can also obtain other related drawings based on these drawings without creative work.

FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present invention;

FIG. 2 is a structural block diagram of an electronic device provided by an embodiment of the present invention;

FIG. 3 is a schematic flowchart of a training set processing method for classification provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of a text sample set in an embodiment of the present invention;

Fig. 5 is a schematic diagram of prediction classification provided by an embodiment of the present invention;

FIG. 6 is a schematic flowchart of step S302 according to an embodiment of the present invention;

Fig. 7 is a schematic diagram of sample knowledge corresponding to each sample provided by the embodiment of the present invention;

FIG. 8 is a schematic flowchart of step S304 provided by an embodiment of the present invention;

Fig. 9 is a schematic diagram of analysis of category disambiguation results provided by the embodiment of the present invention;

Fig. 10 is a functional block diagram of a training set processing device for classification provided by an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. The components of the embodiments of the invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations.

Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

In the description of the present invention, it should be noted that if the orientation or positional relationship indicated by the terms "upper", "lower", "inner" and "outer" appear, it is based on the orientation or positional relationship shown in the drawings, or It is the orientation or positional relationship that the invention product is usually placed in use, and it is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation , and therefore cannot be construed as a limitation of the present invention.

In addition, terms such as "first" and "second" are used only for distinguishing descriptions, and should not be understood as indicating or implying relative importance.

It should be noted that, in the case of no conflict, the features in the embodiments of the present invention may be combined with each other.

Classification tasks are very useful in many scenarios, such as the intention recognition of intelligent customer service, the recognition of flowers, plants and animals in nature, etc. Take the text classification task as an example, please refer to Figure 1, which is an application scenario provided by the embodiment of the present invention A schematic diagram, the scene includes: a terminal device 11 and a server 12 . Wherein, various application programs may be installed in the terminal device 11 . After the communication connection is established between the application program of the terminal device 11 and the server 12 through the communication network, the client of the terminal device 11 can send the text to be recognized to the server 12, and the server 12 performs classification to obtain the classification result, and then sends the classification result to the server 12. Application program for terminal equipment 11.

Wherein, the terminal device 11 may be, but not limited to, a computer device such as a personal computer, a notebook computer, a smart phone, a tablet computer, and a smart wearable device with an information collection function.

The server 12 can be realized by an independent server or a server cluster or a distributed system composed of multiple servers, and can also provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware Cloud server for basic cloud computing services such as domain name service, security service, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platforms. The terminal device 11 and the server 12 may be connected directly or indirectly through wired or wireless communication, which is not limited in this application.

The terminal device 11 and the server 12 can be connected through a communication network, and the communication network can be a wireless fidelity (wireless fidelity, wifi) hotspot network, a bluetooth (bluetooth, BT) network or a near field communication technology (near field communication, NFC) Short-distance communication network such as network, can also be the third-generation mobile communication technology (3rd-generation wireless telephone technology, 3G) network, the fourth-generation mobile communication technology (the4th generation mobile communication technology, 4G) network, the fifth-generation mobile communication technology ( 5th-generation mobile communication technology (5G) network, future evolution of public land mobile network (public land mobile network, PLMN) or the Internet, etc.

Continuing to refer to FIG. 1 , at present, before performing a classification task, it is necessary to use a pre-labeled training set to train a classification model, and then use the trained classification model to perform classification. It can be seen that the classification effect has a great relationship with the data quality of the training set. When there are errors in the labeled samples of the training set, for example, some samples in the training set are labeled incorrectly, or there are conflicts between certain categories, the performance of the classification model Will be affected, seriously affecting the classification effect.

Therefore, the embodiment of the present invention provides a training set processing method, which can firstly perform data cleaning on the training set to realize knowledge disambiguation. The "disambiguation" involved here means eliminating sample labeling errors and eliminating category confusion problems, and will get The results are fed back to the operator, and the operator adjusts the samples or categories to achieve the effect of pre-cleaning the training set.

Please refer to FIG. 2 first. FIG. 2 is a structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device can be used to execute the training set processing method for classification provided by the embodiment of the present invention.

As shown in FIG. 2, the electronic device 200 includes a memory 201, a processor 202, and a communication interface 203, and the memory 201, the processor 202, and the communication interface 203 are electrically connected to each other directly or indirectly to realize data transmission or interaction. . For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

The memory 201 can be used to store software programs and modules, such as the instructions/modules of the training set processing device 400 for classification provided in the embodiment of the present invention, which can be stored in the memory 201 in the form of software or firmware (firmware) or solidified in the electronic device In the operating system (operating system, OS) of 200 , the processor 202 executes various functional applications and data processing by executing software programs and modules stored in the memory 201 . The communication interface 203 can be used for signaling or data communication with other node devices.

Wherein, memory 201 can be but not limited to, random access memory (Random Access Memory, RAM), read-only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-OnlyMemory, PROM), erasable Read-only memory (Erasable Programmable Read-Only Memory, EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc.

The processor 202 may be an integrated circuit chip with signal processing capabilities. The processor 202 can be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (NetworkProcessor, NP), etc.; it can also be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

It can be understood that the structure shown in FIG. 2 is only for illustration, and the electronic device 200 may also include more or less components than those shown in FIG. 2 , or have a configuration different from that shown in FIG. 2 . Each component shown in FIG. 2 may be implemented by hardware, software or a combination thereof.

The electronic device shown in FIG. 2 is used as the execution subject below to introduce in detail the training set processing method for classification provided by the embodiment of the present invention. Please refer to FIG. 3, which shows the training for classification provided by the embodiment of the present invention. A schematic flowchart of a set processing method, the method comprising:

S301: Obtain a classification sample set; the classification sample set contains multiple samples and a classification label corresponding to each sample; a classification label corresponds to at least one sample;

S302: Determine each predicted classification corresponding to each sample, the probability of each predicted classification, and the disambiguation score corresponding to each sample; wherein, the disambiguation score represents the error degree of the classification label corresponding to the sample;

S303: For each sample, determine the most similar sample corresponding to each sample from each sample with the target prediction classification corresponding to the sample, and the probability of the target prediction classification is the largest;

S304: Based on the disambiguation score, determine the most similar predicted category corresponding to each category label, and based on the most similar predicted category, determine the first target sample whose predicted category is the most similar predicted category from all samples with the category label, and From all samples with the most similar predicted classification, determine a second target sample whose predicted classification is the classification label;

Among them, the first target sample and the second target sample are used to indicate the combination strategy for the classification label and the most similar predicted classification, or indicate the implementation of the correction strategy for the first target sample and/or the second target sample; the most similar sample is used for Indicates that the sample is corrected.

In the above-mentioned training set processing method, the classification sample set is first obtained, and then the category of each sample and the probability of each category are predicted. Since each sample has a correct classification label, then for each category, it can be obtained from the Among the prediction results of the samples with this category, determine the predicted category with the largest conflict with the category, and the first target sample corresponding to the predicted category, and from each sample with the predicted category, determine that the predicted result is the category By analyzing the first target sample and the second target sample, it can be determined whether the categories need to be merged or whether the samples need to be adjusted. It is also possible to determine the most similar sample corresponding to each sample, and then determine whether it is necessary to The category of samples is adjusted, and the whole process can quickly identify labeling errors and labeling confusion, which improves the speed of sample cleaning.

The above step S301 to step S304 will be described in detail below.

In step S301, a classification sample set is obtained.

In this embodiment, the classification sample set includes a plurality of samples and a classification label corresponding to each sample; a classification label corresponds to at least one sample;

As an optional implementation manner, the samples in the classification sample set may be corpus, text, image, etc., which are not limited here.

Taking the sample as text as an example, please refer to Fig. 4. Fig. 4 is a schematic diagram of a text sample set in the embodiment of the present invention, wherein sentence is a sample, classify is the classification and label corresponding to the sample, and split_num is the sample identifier. The implementation of the present invention The sample segmentation in the example identifies the data basis for subsequent cross-training on the classification sample set.

As an optional implementation, the background crawling of the application software can be performed through data crawling, or the user's behavior records in the application software can be recorded in real time, and the text entered by the user can be extracted to form a classification sample set, wherein, The application software may be, but not limited to, a media application, a game application, a shopping application, etc., which are not limited here.

It is understandable that the classification labels are manually marked, and the classification labels corresponding to each sample may match the actual category of the sample, that is, the classification label is correctly classified, or the classification label does not match the actual category of the sample, that is, the classification label is not For correct classification, the embodiments of the present invention can assist operators in determining mislabeled samples and confusing categories.

In step S302, each predicted category corresponding to each sample, the probability of each predicted category, and the disambiguation score corresponding to each sample are determined.

In this embodiment, each sample can be predicted through the classification model in the embodiment of the present invention, and the predicted classification corresponding to each sample and the probability of the predicted classification can be obtained. It can be understood that the predicted classification referred to by the probability is the sample The probability of the corresponding class label.

The above disambiguation score can be used to roughly estimate the error degree of the classification label corresponding to the sample. In this embodiment, the disambiguation score is the maximum value of the maximum misclassification probability and the confusion probability, wherein the maximum misclassification probability is: different from the classification label However, for the probability corresponding to the predicted category with the highest probability, the confusion probability is: the difference between the first two largest probabilities in all probabilities. In this way, the embodiment of the present invention obtains the disambiguation score of the sample through the above two indicators, which can not only For labeling errors, it can also be used for confusing labels, which provides a data basis for subsequent identification of whether there are labeling errors and labeling confusion in the sample.

Referring to Fig. 5, Fig. 5 is a schematic diagram of the prediction classification provided by the embodiment of the present invention. In Fig. 5, the TOP2 prediction classification and probability corresponding to a sentence can be seen. The definition is not limited to the display probability for the TOP2 prediction classification, and the prediction results of all samples can be obtained through the above method.

Therefore, this embodiment also provides the implementation of step S302, please refer to FIG. 6, which is a schematic flowchart of step S302 in the embodiment of the present invention:

S302-1: Using the trained classification model, obtain each predicted category of each sample and the probability of each predicted category.

In this embodiment, the classification model can be trained by any existing training method.

As one of the optional implementation manners, the training method of the classification model in this embodiment may include the following steps:

a1: According to the sample identification corresponding to each sample, determine multiple sample subsets and the prediction sample set corresponding to each sample subset from the training set.

Among them, the union of each sample subset and the prediction sample set corresponding to the sample subset is a classification sample set; in a sample subset, the sample identification of each sample is the same as the remainder obtained by dividing the number of sample subsets, and The remainder is different from the remainder corresponding to each sample in the forecast sample set.

In this embodiment, the sample ID of each sample is divided by the number of sample subsets to obtain the remainder corresponding to each sample, and then the samples with the same remainder form the sample subset, and then the samples with the other remainder form the sample The prediction sample set corresponding to the subset.

For the convenience of understanding, assuming that the number of sample subsets is 3, and the remainder is set to 0, 1, 2 to divide the classification sample set into 3 sample subsets, then you can divide "split_num" by 3 to obtain samples with a remainder of 1, 2 Form the first sample subset, denoted as S ₁ , 2, then the samples with a remainder of 0 are used as the prediction sample set of S _{1, 2} ; the samples with a remainder of 0, 1 form the second sample subset, denoted as S _{0 , 1} , then the sample with a remainder of 2 is used as the prediction sample set of the S _{0 ,} 1 sample subset; As the prediction sample set of S _0,2 .

That is to say, in the actual implementation process, the number N of sample subsets can be determined first, and then N different residues consistent with the number of sample subsets can be determined, and then in the process of determining the sample subsets, first The samples with N-1 remainders form a sample subset, and the samples with another remainder form a prediction sample set of the sample subset, and so on, to obtain multiple sample subsets and each sample subset corresponds to a prediction sample set.

a2: Use each sample subset to train the classification model in turn, and use the trained classification model to predict the prediction sample set corresponding to the sample subset, and obtain each prediction category of each prediction sample in the prediction sample set, and each Probability of predicted class.

S302-2: For each sample, based on the comparison result of the predicted classification and the classification label corresponding to the sample, determine the maximum misclassification probability and confusion probability corresponding to the sample.

The purpose of comparing the predicted classification with the classification label corresponding to the sample is to determine whether there is a prediction classification with the same classification label. Different comparison results have different guidance directions for the subsequent determination of the maximum misclassification probability.

In this embodiment, the maximum misclassification probability refers to the probability of misidentifying the sample category (that is, it is different from the classification label). For example, assuming a class A sample, the predicted classification results include class A, class B and class C, then , both class B and class C can be regarded as the misclassification of class A samples, and the probability of misclassification with the highest probability is taken as the maximum misclassification probability, that is, the probability of class B.

Confusion probability is the probability of conflicting categories in each predicted classification of a sample. For example, continue to use the above example to illustrate that the relationship between the corresponding probabilities of the above-mentioned predicted results of class A, class B and class C is P _A > P _B >P _C , then the confusion probability is the difference between the probability of class A and class B, and the degree of confusion between class A and class B can be measured by the confusion probability.

Therefore, the embodiment of the present invention provides an implementation manner of step S302-2, namely:

b1: Determine the first maximum probability and the second maximum probability in order of probability from large to small, and use the difference between the first maximum probability and the second maximum probability as the confusion probability.

b2: If the predicted classification corresponding to the first maximum probability is consistent with the classification label, then the second maximum probability is used as the maximum misclassification probability.

b3: If the predicted classification corresponding to the first maximum probability is inconsistent with the classification label, then the first maximum probability is used as the maximum misclassification probability.

S302-3: Determine the maximum value of the maximum misclassification probability and the confusion probability as the disambiguation score corresponding to the sample.

Through the maximum misclassification probability and confusion probability obtained above, the disambiguation score corresponding to each sample can be determined, and then the output category disambiguation analysis results and sample disambiguation analysis results can be sorted out. The following will first introduce the sample disambiguation analysis results.

Please refer to FIG. 7. FIG. 7 is a schematic diagram of the sample knowledge corresponding to each sample provided by the embodiment of the present invention. It can be seen from FIG. 7 that for each sample, the embodiment of the present invention can obtain the disambiguation score corresponding to the sample. And the most similar category corresponding to the category of the sample, the probability of the most similar category, the probability of confusion, etc. Based on these sample knowledge, the embodiment of the present invention can continue to dig deeper to obtain categories that may have labeling errors or are prone to confusion.

In step S303, for each sample, determine the most similar sample corresponding to the sample from each sample with the target prediction classification corresponding to the sample, and the probability of the target prediction classification is the largest;

In this embodiment, each sample corresponds to multiple predicted categories, and the predicted category corresponding to the maximum probability is used as the target predicted category. It should be noted that the predicted category corresponding to the maximum probability may be the same as or different from the category label of the sample , so the most similar samples can be determined based on two different situations.

The first case: if the target prediction classification is consistent with the classification label corresponding to the sample, the output most similar sample is empty;

The second case: if the target prediction classification is inconsistent with the classification label corresponding to the sample, extract all candidate samples with the target prediction classification from the classification sample set, calculate the similarity between all candidate samples and the sample, and compare the candidate with the largest similarity The sample is determined to be the most similar sample.

In this embodiment, all the samples in the classification sample set can be sorted in descending order of the disambiguation scores, and then the most similar samples of each sample can be displayed in sequence, so as to facilitate the analysis of possible reasons for prediction errors.

The results of category disambiguation analysis are introduced first.

In step S304, based on the disambiguation score, determine the most similar prediction classification corresponding to each classification label, and based on the most similar prediction classification, from all samples with classification labels, determine the prediction classification as the first target of the most similar prediction classification sample, and from among all samples with the most similar predicted class, determine a second target sample whose predicted class is the class label.

In this embodiment, a classification label corresponds to at least one sample, that is, assuming that there are M samples in the classification sample set, and the corresponding number of categories is N, then N and M satisfy the relationship that N is less than or equal to M, then for each The embodiment of the present invention can find the most similar category with the largest conflict. The largest conflict can be understood as the two categories cannot be easily distinguished. The reason for this situation may be that the granularity of category division is too small.

Therefore, the embodiment of the present invention finds out the most similar category corresponding to each category, and then finds out the corresponding samples of this category and the most similar category, and checks these samples to determine whether it is necessary to perform category merging or whether it is necessary to change the sample. Classification labels.

As an optional implementation manner, the above step S304 can be implemented in the manner shown in FIG. 8 , and FIG. 8 is a schematic flowchart of step S304 provided by an embodiment of the present invention:

S304-1. Determine samples to be confirmed with the same predicted classification from all samples with classification labels.

S304-2. Determine the sum of the disambiguation scores of all samples to be confirmed with the same predicted category, and determine the corresponding predicted category when the sum of the disambiguated scores is the largest as the most similar predicted category;

S304-3. Determine all unconfirmed samples corresponding to the most similar predicted classification as first target samples, and determine a second target sample whose predicted classification is a classification label from all samples with the most similar predicted classification.

In this embodiment, since the predicted classification of each sample is obtained in advance, for each classification label, the sample predicted to be classified as the classification label can be determined as the second target sample from all samples. And since there are multiple samples corresponding to each classification label, the sample whose predicted classification is the most similar to the predicted classification can be determined as the first target sample.

In this embodiment, the first target sample and the second target sample are used to indicate to perform a merge strategy on the classification label and the most similar predicted classification, or to indicate to perform a correction strategy on the first target sample and/or the second target sample.

For example, based on the above results, the operator can find out the similarity between two categories of wrong predictions (ie, the first target sample and the second target sample), and if they feel that there is a big conflict, they can consider merging categories, or make corresponding sample adjustments.

For example, the first target sample is marked as class A, but the classification model predicts that the first target sample is class B, which indicates that it is reasonable to classify it as class A or class B for the same text, then the operator at this time It can be determined whether the first target sample needs to be classified, or judge whether the first target text needs to change the classification label so that it can be completely distinguished from other categories, and the same is true for the second target category.

For the convenience of understanding, assume that one of the classification labels is "why the membership rights cannot be applied", and there are 10 samples with this classification label, 5 samples of the predicted classification, and 3 samples of the predicted classification S2, and we get There are 2 samples in the predicted category S3. Since the disambiguation score of each sample has been obtained in the foregoing content, the disambiguation score of all samples corresponding to each category is added to obtain the disambiguation score corresponding to this category. and, that is, add the disambiguation scores of the 5 samples corresponding to S1, add the disambiguation scores of the 3 samples corresponding to S2, add the disambiguation scores of the 2 samples corresponding to S3, and finally get each The sum of the disambiguation scores of each prediction category can be understood as: the weight of this category that needs to be disambiguated, and the category corresponding to the maximum sum of the disambiguation scores is determined as the most similar prediction category of "why the membership benefits cannot be applied".

Please refer to Fig. 9. Fig. 9 is a schematic diagram of the category disambiguation result analysis provided by the embodiment of the present invention. It can be seen that for each classification, the embodiment of the present invention can find out the most similar classification to it, and then classify the samples As a result, samples that may have conflicts were obtained, and the results shown in Figure 9 were displayed to the operator for knowledge disambiguation, data cleaning, and labeling accuracy were improved.

Therefore, in the embodiment of the present invention, after obtaining the most similar sample corresponding to each sample, the most similar classification corresponding to each classification, and the sum of the first target sample and the second target sample that may have conflicts, in order to facilitate the analysis of the operator, the The above results can be displayed as follows:

According to the order of the sum of the disambiguation scores from large to small, the classification label, the number of samples corresponding to the classification label, the most similar predicted classification corresponding to the classification label, and the most similar predicted classification corresponding to the first target sample and the first target sample are displayed in sequence The probability of the second target sample and the probability of the classification label corresponding to the second target sample;

In descending order of the disambiguation score, each sample, the classification label corresponding to the sample, the predicted classification, the disambiguation score, and the most similar sample are displayed in turn.

The training set processing method for classification provided in the embodiment of the present invention can be implemented in a hardware device or implemented in the form of a software module. When the training set processing method for classification is implemented in the form of a software module, the embodiment of the present invention also Provide a training set processing method device for classification, please refer to Figure 10, Figure 10 is a functional block diagram of a training set processing device for classification provided by an embodiment of the present invention, the training set processing device for classification 400 Can include:

An acquisition module 410, configured to acquire a classification sample set; the classification sample set includes a plurality of samples and classification labels corresponding to each of the samples; one classification label corresponds to at least one of the samples;

A determining module 420, configured to determine each predicted classification corresponding to each of the samples, the probability of each of the predicted classifications, and the disambiguation score corresponding to each of the samples; wherein the disambiguation score represents that the sample corresponds to The error score of the classification label of ;

The determination module 420 is further configured to, for each sample, determine the most similar sample corresponding to each sample from each sample with the target prediction classification corresponding to the sample, and the probability of the target prediction classification Maximum; the probability of the target prediction classification is maximum;

The determination module 420 is further configured to determine the most similar predicted category corresponding to each of the category labels based on the disambiguation score, and based on the most similar predicted category, from all samples with the category label, determining a first target sample whose predicted class is the most similar predicted class, and determining a second target sample whose predicted class is the class label from all samples having the most similar predicted class;

Wherein, the first target sample and the second target sample are used to indicate that a merging strategy is performed on the classification label and the most similar predicted classification, or indicate that the first target sample and/or the second Two target samples implement a correction strategy; the most similar sample is used to indicate to correct the sample.

It can be understood that the acquisition module 410 and the determination module 420 can coordinately execute each step in FIG. 3 to achieve corresponding technical effects.

In an optional embodiment, the determination module 420 is specifically configured to: use the trained classification model to obtain the respective predicted categories of each sample and the probability of each predicted category; for each sample , based on the comparison result of the predicted classification and the classification label corresponding to the sample, determine the maximum misclassification probability and confusion probability corresponding to the sample; the maximum misclassification probability and the maximum value of the confusion probability, Determine the disambiguation score corresponding to the sample.

In an optional embodiment, the determining module 420 is specifically configured to: determine a plurality of sample subsets and a prediction sample corresponding to each of the sample subsets from the training set according to the sample identification corresponding to each sample set; wherein, the union of each of the sample subsets and the predicted sample sets corresponding to the sample subsets is the classification sample set; in one of the sample subsets, the sample identification of each of the samples The remainder obtained by dividing the number of sample subsets is the same, and the remainder is different from the remainder corresponding to each prediction sample in the prediction sample set; each of the sample subsets is used in turn to train the classification model, and using the training The subsequent classification model predicts the prediction sample set corresponding to the sample subset, and obtains the respective prediction classifications of each prediction sample in the prediction sample set and the probability of each prediction classification.

In an optional implementation, the determination module 420 is specifically configured to: determine the first maximum probability and the second maximum probability in descending order of probability, and calculate the first maximum probability and the second maximum probability The difference between the probabilities is used as the confusion probability; if the predicted classification corresponding to the first maximum probability is consistent with the classification label, then the second maximum probability is used as the maximum misclassification probability; if the first If the predicted category corresponding to the maximum probability is inconsistent with the category label, the first maximum probability is used as the maximum misclassification probability.

In an optional implementation, the determination module 420 is specifically configured to: determine the samples to be confirmed with the same predicted classification from all samples with the classification label; determine all the samples with the same predicted classification The sum of the disambiguation scores of the samples to be confirmed, and the corresponding prediction classification when the sum of the disambiguation scores is the largest is determined as the most similar prediction classification; all the samples to be confirmed corresponding to the most similar prediction classification , determine as the first target sample, and determine a second target sample whose predicted class is the class label from all samples with the most similar predicted class.

In an optional implementation, the determination module 420 is specifically configured to: if the predicted classification of the target is consistent with the classification label corresponding to the sample, then output the most similar sample as empty; if the predicted classification of the target is consistent with the classification label corresponding to the sample The corresponding classification labels are inconsistent, extract all candidate samples with the target prediction classification from the classification sample set, calculate the similarity between all the candidate samples and the samples, and determine the candidate sample with the largest similarity as The most similar sample.

In an optional embodiment, the device may further include a display module, configured to sequentially display the classification label, the number of samples corresponding to the classification label, the The most similar predicted classification corresponding to the classification label, and the probability of the most similar predicted classification corresponding to the first target sample and the first target sample, the second target sample and the second target sample The probability of the corresponding classification label; in descending order of the disambiguation score, each of the samples, the classification label corresponding to the sample, the predicted classification, the disambiguation score, and the most similar sample are displayed in sequence.

An embodiment of the present invention also provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the training set processing method for classification according to any one of the foregoing implementations is implemented. The computer-readable storage medium may be, but not limited to, various mediums capable of storing program codes such as U disk, mobile hard disk, ROM, RAM, PROM, EPROM, EEPROM, magnetic disk or optical disk.

It should be understood that the devices and methods disclosed in the present invention can also be implemented in other ways. The device embodiments described above are only illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functions and possible implementations of devices, methods and computer program products according to multiple embodiments of the present invention. operate. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more executable instruction. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or action , or may be implemented by a combination of dedicated hardware and computer instructions.

In addition, each functional module in each embodiment of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

The above are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention. It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

Claims (10)

1. A method of training set processing for classification, the method comprising:

obtaining a classification sample set; the classification sample set comprises a plurality of samples and classification labels corresponding to the samples; one classification label corresponds to at least one sample;

determining a respective prediction classification for each of the samples, a probability for each of the prediction classifications, and a disambiguation score for each of the samples; wherein the disambiguation score characterizes a degree of error of the classification label corresponding to the sample;

for each sample, determining a most similar sample corresponding to each sample from the samples having a target prediction classification corresponding to the sample, the probability of the target prediction classification being the maximum;

determining a most similar prediction classification corresponding to each classification label based on the disambiguation score, and determining a first target sample of which the prediction classification is the most similar prediction classification from all samples with the classification labels and a second target sample of which the prediction classification is the classification label from all samples with the most similar prediction classification based on the most similar prediction classification;

wherein the first target sample and the second target sample are used to instruct a merge strategy to be performed on the class labels and on the most similar prediction classes or to instruct a correction strategy to be performed on the first target sample and/or the second target sample; the most similar sample is used to indicate that the sample is corrected.

2. The method of claim 1, wherein determining a predictive classification for each of the samples, a probability of the predictive classification, and a disambiguation score for each of the samples comprises:

obtaining each prediction classification of each sample and the probability of each prediction classification by using the trained classification model;

for each sample, determining the maximum misclassification probability and the confusion probability corresponding to the sample based on the comparison result of the prediction classification and the classification label corresponding to the sample;

and determining the maximum value of the maximum misclassification probability and the confusion probability as the corresponding disambiguation score of the sample.

3. The method of claim 2, wherein obtaining the respective prediction classification corresponding to each of the samples and the probability of each of the prediction classifications using the trained classification model comprises:

determining a plurality of sample subsets and a prediction sample set corresponding to each sample subset from the training set according to the sample identification corresponding to each sample;

wherein each sample subset and the union of the prediction sample sets corresponding to the sample subsets are the classification sample sets; in one of the sample subsets, the remainder obtained by dividing the sample identifier of each sample by the number of the sample subset is the same, and the remainder is different from the remainder corresponding to each predicted sample in the predicted sample set;

and sequentially training a classification model by using each sample subset, and predicting a prediction sample set corresponding to the sample subset by using the trained classification model to obtain each prediction classification of each prediction sample in the prediction sample set and the probability of each prediction classification.

4. The method of claim 2, wherein determining, for each sample, a maximum misclassification probability and a confusion probability for the sample based on the comparison of the predicted classification to the classification label corresponding to the sample comprises:

determining a first maximum probability and a second maximum probability according to the sequence of the probabilities from large to small, and taking the difference between the first maximum probability and the second maximum probability as the confusion probability;

if the prediction classification corresponding to the first maximum probability is consistent with the classification label, taking the second maximum probability as the maximum error classification probability;

and if the prediction classification corresponding to the first maximum probability is inconsistent with the classification label, taking the first maximum probability as the maximum error classification probability.

5. The method of claim 1, wherein determining a most similar prediction classification corresponding to each of the classification tags based on the disambiguation score, and determining a first target sample for which a prediction classification is the most similar prediction classification from all samples having the classification tag and a second target sample for which a prediction classification is the classification tag from all samples having the most similar prediction classification based on the most similar prediction classification comprises:

determining samples to be confirmed having the same prediction classification from all samples having the classification label;

determining the sum of disambiguation scores of all samples to be confirmed with the same prediction classification, and determining the prediction classification corresponding to the maximum sum of the disambiguation scores as the most similar prediction classification;

and determining all samples to be confirmed corresponding to the most similar prediction classification as the first target sample, and determining a second target sample of which the prediction classification is the classification label from all samples with the most similar prediction classification.

6. The method of claim 1, wherein for each sample, determining, for each sample, a most similar sample for each sample from the samples having the target prediction classification to which the sample corresponds comprises:

if the target prediction classification is consistent with the classification label corresponding to the sample, outputting the most similar sample as null;

if the target prediction classification is inconsistent with the classification label corresponding to the sample, extracting all candidate samples with the target prediction classification from the classification sample set, calculating the similarity between all the candidate samples and the sample, and determining the candidate sample with the maximum similarity as the most similar sample.

7. The method of claim 5, further comprising:

sequentially displaying the classification labels, the number of samples corresponding to the classification labels, the most similar prediction classification corresponding to the classification labels, the probability of the most similar prediction classification corresponding to the first target sample and the first target sample, and the probability of the classification labels corresponding to the second target sample and the second target sample according to the sequence that the sum of the disambiguation scores is from large to small;

and sequentially displaying each sample, the classification label corresponding to the sample, the prediction classification, the disambiguation score and the most similar sample according to the sequence of the disambiguation score from large to small.

8. A training set processing apparatus for classification, comprising:

the acquisition module is used for acquiring a classification sample set; the classification sample set comprises a plurality of samples and classification labels corresponding to the samples; one classification label corresponds to at least one sample;

a determining module for determining respective prediction classifications corresponding to each of the samples, a probability of each of the prediction classifications, and a disambiguation score corresponding to each of the samples; wherein the disambiguation score characterizes an error score of the classification label corresponding to the sample;

the determining module is further configured to determine, for each sample, a most similar sample corresponding to each sample from samples having a target prediction classification corresponding to the sample, where the probability of the target prediction classification is the maximum; the probability of the target prediction classification is maximum;

the determining module is further configured to determine a most similar prediction classification corresponding to each of the classification labels based on the disambiguation score, and determine a first target sample of which the prediction classification is the most similar prediction classification from all samples having the classification label and a second target sample of which the prediction classification is the classification label from all samples having the most similar prediction classification based on the most similar prediction classification;

wherein the first target sample and the second target sample are used to instruct a merge strategy to be performed on the class labels and on the most similar prediction classes or to instruct a correction strategy to be performed on the first target sample and/or the second target sample; the most similar sample is used to indicate that the sample is corrected.

9. An electronic device comprising a processor and a memory, the memory storing a computer program executable by the processor, the processor being configured to execute the computer program to perform the method of any of claims 1 to 7.

10. A storage medium on which a computer program is stored, which computer program, when being executed by a processor, carries out the method according to any one of claims 1 to 7.

Images