KR20230064304A - Apparatus for automatic labeling and method for self-learning of language model using the same - Google Patents

Abstract

As a method of labeling spoken sentences by an automatic labeling apparatus, a classifier is trained to receive training data labeled with speech intentions and output the speech intentions from the learning sentences included in the respective training data. When learning data is input to the learned classifier, a keyword phrasebook for the learning sentence is generated, and a speech sentence in which the speech intent is not labeled is received, a plurality of candidate labels for the speech sentence and the speech sentence are obtained using the learned classifier. The utterance intention of the uttered sentence is output using the keyword phrasebook generated from .
At this time, the classifier receives a language model learned to extract speech intention and keywords from the learning data and spoken sentences, and the keywords extracted by the language model, extracts a plurality of words related to the keyword from the learning data and spoken sentences, and It also includes an attention mechanism that generates keyword phrasebooks.

Description

Automatic labeling device and method for labeling spoken sentences using the same {Apparatus for automatic labeling and method for self-learning of language model using the same}

The present invention relates to an automatic labeling apparatus for automatically labeling unlabeled sentences and learning a deep learning model to improve the accuracy of pseudo labeling of semi-supervised learning, and a labeling method for spoken sentences using the same.

Recently, in the field of natural language processing, research to solve text classification problems has been actively conducted.

Language models such as BERT (Bidirectional Encoder Representations from Transformers) are used by domestic and foreign companies for various services. Research to build deep learning model learning data using a teacher-student learning method that helps light learning with a large-capacity model such as BERT using a large amount of unlabeled unstructured data is receiving a lot of attention. there is.

In text classification, labeling of unstructured data takes a lot of time and cost. In addition, since sufficient learning data for learning is not secured, related technologies such as quasi-supervised learning and data augmentation techniques are being studied to solve the limited learning data problem. Representatively, there is a pseudo labeling technique that expands data by automatically labeling unlabeled data using a deep learning model trained with a small amount of training data.

Since the pseudo-labeling technique is biased only to the predicted value of the deep learning model trained with a small amount of training data, there is a problem in that the performance of the model deteriorates when an error occurs in the pseudo-labeling process. In particular, when the data distribution is very imbalanced, such as when the training data is short or the number of labels is 100 or more, there is a problem in that more errors occur in the pseudo-labeling process and the performance of the model deteriorates significantly.

Therefore, the present invention utilizes a self-attention score calculated through a deep learning model, a keyword phrasebook generated by an external attention score calculated by an attention mechanism, and a plurality of top model predictions predicted through a language model, and uses a large number of unlabeled An automatic labeling device for automatically labeling positive data and self-learning a language model with training data and a method for labeling spoken sentences using the same are provided.

As a method of labeling a spoken sentence by an automatic labeling device, which is one feature of the present invention for achieving the technical problem of the present invention,

learning a classifier to receive learning data labeled with speech intention and output the speech intention from a learning sentence included in each of the learning data; Generating a keyword phrasebook, receiving a spoken sentence whose speech intent is not labeled, and using a plurality of candidate labels for the spoken sentence and a keyword phrasebook generated from the spoken sentence using a learned classifier, and outputting a speech intent of a spoken sentence, wherein the classifier receives a language model learned to extract a speech intent and a keyword from the learning data and the spoken sentence, and the keyword extracted by the language model, and performs the learning and an attention mechanism that extracts a plurality of words related to the keyword from data and the spoken sentence and generates a keyword phrasebook together with the keyword.

The outputting of the utterance intention may include comparing a first candidate label having the highest prediction reliability among a plurality of candidate labels generated from the utterance sentence with a preset threshold, and the prediction reliability of the first candidate label. and outputting the first candidate label as a correct answer of the spoken sentence and setting the first candidate label as a correct answer label for the spoken sentence when is greater than the threshold value.

If the prediction reliability of the first candidate label is less than the threshold value, any one of the plurality of candidate labels excluding the first candidate label is set as the correct answer label based on the keyword phrasebook generated from the spoken sentence. A setting step may be further included.

The outputting of the speech intention may include extracting, as a keyword of the spoken sentence, a word for which the self-attention score is highest in the spoken sentence by a self-attention mechanism included in the language model; The method may include extracting at least one related word related to the keyword extracted by the mechanism, and generating the keyword phrasebook including the keyword and the at least one related word.

After outputting the utterance intention, re-learning the classifier using a utterance sentence for which the correct answer label is set may be included.

As an automatic labeling device for labeling a speech intention in a spoken sentence, which is another feature of the present invention for achieving the technical problem of the present invention,

A memory containing at least one instruction and storing labeled training data, an interface for receiving an unlabeled utterance sentence uttered by a speaker, and

and a processor, wherein the processor extracts a learning sentence and an utterance intent for the learning sentence from the training data, uses the learning sentence and the utterance intent to train a classifier, and uses the learned classifier to determine the learning sentence. A phrasebook of keywords is created, and the spoken sentence is input into the learned classifier to label the spoken sentence with an intention for the spoken sentence.

The processor calculates prediction reliability of speech intent from the spoken sentence, extracts a plurality of candidate labels, compares a first candidate label having the highest prediction reliability among the extracted candidate labels with a preset threshold, and compares the first candidate label with a preset threshold. 1 It can be determined whether the candidate label is the correct label.

The processor, when the prediction reliability of the first candidate label is greater than the threshold value, outputs the first candidate label as a correct answer of the spoken sentence and sets it as a correct answer label for the spoken sentence;

If the prediction reliability of the first candidate label is less than the threshold value, any one of the plurality of candidate labels excluding the first candidate label is set as the correct answer label based on the keyword phrasebook generated from the spoken sentence. can be set

The processor extracts a word for which the self-attention score is highest in the spoken sentence as a keyword of the spoken sentence, extracts at least one related word related to the keyword, and extracts the keyword and the at least one related word. It is possible to generate the keyword phrasebook including.

According to the present invention, based on the self-attention score of the deep learning model and the attention score calculated through the attention mechanism, a phrasebook for keywords extracted from sentences and a plurality of upper model predicted values predicted through a language model are used, By providing the model to proceed with self-learning, it is possible to improve the prediction performance of the language model.

In addition, since data can be automatically labeled using a very small amount of training data and a large amount of unstructured data, learning data can be easily constructed.

1 is a structural diagram of an automatic labeling device according to an embodiment of the present invention.
2 is a flowchart of a method of labeling a spoken sentence by an automatic labeling apparatus according to an embodiment of the present invention.
3 is an exemplary view of a keyword phrasebook generated according to an embodiment of the present invention.
4 is an exemplary view of a learning procedure of a general classifier.
5 is an exemplary view of a learning procedure of a classifier according to an embodiment of the present invention.
6 is a structural diagram of a computer system according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated.

Hereinafter, an automatic labeling apparatus according to an embodiment of the present invention and a method for labeling spoken sentences using the same will be described in detail with reference to the drawings.

1 is a structural diagram of an automatic labeling device according to an embodiment of the present invention.

As shown in FIG. 1 , the automatic labeling apparatus 100 includes a classifier composed of a language model 110 such as BERT and an attention mechanism 120 .

When a small amount of labeled training data is input to the language model 110, a label set in the training data is output as a correct answer. In addition, when unlabeled data is input, the language model 110 finds a correct label for the corresponding data, automatically labels it, and outputs the labeled data.

That is, the language model 110 receives a sentence uttered by a speaker as data, and predicts the reliability of the corresponding sentence for each utterance intention. A method in which the language model 110 predicts reliability for each utterance intent in a sentence is a known technique, and a detailed description thereof is omitted in an embodiment of the present invention.

The language model 110 outputs a correct answer label according to the prediction reliability of the labeled training data. In addition, the language model 110 automatically labels unlabeled bulk data and outputs the labeled data.

To this end, the language model 110 is trained to output a label having the highest prediction reliability value in an input sentence using a small amount of training data in which labels are set.

At this time, the language model 110 extracts, as candidate labels, labels whose prediction reliability values are calculated as the top n values in the input sentence. In an embodiment of the present invention, extracting top 5 labels (top 1 to top 5) as candidate labels will be described as an example.

The language model 110 uses the keyword phrasebook output through the attention mechanism 120 to be described later, input data from which the keyword phrasebook is extracted, and candidate labels, and learns again to find the correct answer from the input data. Then, the retrained language model 110 automatically labels large-volume data for which no label has been set.

The language model 110 includes a self-attention mechanism 111 therein. When labeled training data is input, the self-attention mechanism 111 extracts a representative word representing the input training data as a keyword. A method of extracting a representative keyword from an input sentence by the self-attention mechanism 111 is a known technique, and is not limited to any one method in an embodiment of the present invention.

The attention mechanism 120 extracts at least one related word associated with the keyword using the keyword extracted by the self-attention mechanism 111 and the labeled training data from which the keyword is extracted. Since the attention mechanism 120 may use various methods to extract at least one related word associated with a keyword, the present embodiment is not limited to any one method.

Then, the attention mechanism 120 creates a keyword phrasebook by expressing the keyword and at least one extracted related word as an idiom. This is to solve labeling errors that may occur when related words are used in ambiguous expressions.

This is a known technique in which the attention mechanism 120 extracts related words related to a representative keyword, and is not limited to any one method in an embodiment of the present invention. In addition, the attention mechanism 120 arranges the words in the order of “[keyword, related word]” to generate an idiom, and the generated at least one idiom can be generated as a keyword phrasebook in various ways. And, if there is another related word related to the related word, the attention mechanism 120 may generate a keyword phrasebook in the form of [keyword, related word 1, related word 2].

Next, a method of automatically labeling unlabeled sentences by the automatic labeling apparatus 100 will be described with reference to FIG. 2 .

2 is a flowchart of a method of labeling a spoken sentence by an automatic labeling apparatus according to an embodiment of the present invention.

As shown in FIG. 2 , when labeled training data is input to the automatic labeling apparatus 100, the classifier trains a language model with the labeled training data (S100). That is, when a sentence corresponding to training data is input, the language model is trained to output a correct answer set as a label of the corresponding sentence.

The classifier generates a keyword phrasebook for the training data using the language model trained in this way (S110). That is, a keyword phrasebook is created by including the keywords extracted from the learning data in the self-attention mechanism 111 included in the language model 110 and the related words extracted for the corresponding keywords in the attention mechanism 120.

Here, the self-attention mechanism 111 extracts, as a keyword, a word having the highest reliability value in a sentence corresponding to the learning data. In addition, the attention mechanism 120 extracts as many related words as the number of intentions related to the keyword extracted by the self-attention mechanism 111 from the corresponding sentence.

For example, it is assumed that "Tell me the mobile phone bill" is input as training data. At this time, it is assumed that "wireless charge inquiry" is set as the correct answer intention of the learning data.

The self-attention mechanism 111 of the language model 110 extracts "mobile phone" having the highest self-attention score in "tell me the mobile phone bill" as a keyword. Then, the attention mechanism 120 extracts "fee" and "inform" as related words related to the keyword "mobile phone". Accordingly, the attention mechanism 120 generates a keyword phrasebook including [mobile phone, fee, inform].

In this way, it is assumed that after the language model 110 learns with the labeled training data or generates a keyword phrasebook for the training data, the speaker utters an arbitrary sentence, that is, a spoken sentence and receives the spoken sentence (S120). At this time, it is assumed that the spoken sentence does not have a label set.

The language model 110 pre-learned using the learning data calculates prediction reliability from the spoken sentence, extracts a plurality of candidate labels, and generates a keyword phrasebook for the spoken sentence (S130). In an embodiment of the present invention, extraction of five candidate labels (Top 1 to Top 5) will be described as an example.

The language model 110 compares the prediction reliability of the label predicted by the first candidate label with a preset threshold in the process of calculating the prediction reliability value from the spoken sentence (S140 and S150). If the prediction reliability of the first candidate label is greater than the threshold value, the language model 110 sets the first candidate label as the correct answer label of the spoken sentence (S160).

However, if the prediction reliability of the first candidate label is less than the threshold value, the language model 110 uses the keyword phrasebook generated in step S130 to return to the first candidate among the plurality of candidate labels extracted. A correct answer label is searched from the fifth candidate labels among the labels (S170). That is, when the prediction reliability of the first candidate label predicted by the language model 110 is less than the threshold value, the idiom according to the spoken sentence is searched again from the idiom books from the first candidate labels to the fifth candidate labels.

After finding the correct answer label in this way, the classifier is retrained using the utterance sentence in which the correct answer label is set.

An example of generating a keyword phrasebook in step S110 or step S130 in the process of setting labels on large amount of unlabeled data by the language model 110 according to the above-described procedure will be described with reference to FIG. 3 .

3 is an exemplary view of a keyword phrasebook generated according to an embodiment of the present invention.

The language model 110, that is, the attention mechanism combined with BERT assigns a high score to a key word to classify a class and proceeds with learning. When the language model 110 predicts a spoken sentence such as “I want to know the price of a mobile phone now” using the learned attention mechanism, it assigns a high attention score to a word that is key to predicting a label.

That is, as shown in FIG. 3 , it can be seen that the language model 110 assigns a high attention score of 0.5 to “mobile phone”.

In this way, when attention scores are assigned to each word in the utterance sentence, the language model 110 checks the self-attention score included in the language model 110 based on the word with the high attention score, and then selects a higher score based on the attention score. Extract N related words. 3 shows an example in which three words are extracted as related words.

That is, the language model 110 that extracts “mobile phone,” the word with the highest score in the attention mechanism, as a keyword extracts the top three words from the self-attention score based on “mobile phone,” [now, mobile phone, rate ], that is, constitutes a keyword phrasebook.

As semi-supervised learning progresses and the amount of training data increases, the performance of calculating the attention score and self-attention score used to construct the keyword phrasebook of the language model 110 improves, and a phrasebook of higher quality is built. By repeating this procedure, a keyword phrasebook for each of the five candidate labels extracted from the spoken sentence is automatically built.

That is, when learning the language model 110 with labeled training data, phrasebooks are retrieved for each intent. After labeling unlabeled data by the method according to an embodiment of the present invention, the language model 110 is re-learned by using the labeled data as training data.

If the re-learning process proceeds in this way, the prediction performance of the language model 110 is improved. If this process is repeated, labeling of data that was not initially labeled becomes accurate, and a high-quality phrasebook is built.

Next, an example of setting a label for large-volume data that is not labeled by a conventional language model and an example of setting a label according to an embodiment of the present invention will be described with reference to FIGS. 4 and 5 .

4 is an exemplary view of a learning procedure of a general classifier.

As shown in FIG. 4, conventionally, when the classifier BERT is trained with a small amount of labeled training data, a large amount of unlabeled data is labeled with the learned classifier. Then, re-learning of the classifier is performed by adding the designated labeling to the training data.

At this time, the classifier trained with a small amount of labeled data has poor performance. And, if the intent of the data or the number of data is very large, the classifier may not properly predict the top 1 label.

If, as shown in FIG. 4, it is assumed that intentions are classified into wired/wireless specific words, the conventional learned classifier has a disadvantage in that it is difficult to clearly provide prediction intentions as correct answers by distinguishing them.

5 is an exemplary view of a learning procedure of a classifier according to an embodiment of the present invention.

Previously, only the top1 result of the deep learning model was used to label unlabeled data, so if the amount of data used to train the language model is small, the performance of the language model is not high, and the label is wrong for unlabeled data. There is a downside to specifying .

However, as shown in FIG. 5 , in the embodiment of the present invention, a keyword phrasebook is built using the self-attention score of the language model itself and the attention score of an external language model. In addition, it is possible to select the correct answer from the candidate labels by using the candidate labels predicted as top2 to top5, not only the top1 prediction value of the language model, and using the generated keyword phrasebook. Therefore, in the embodiment of the present invention, when the candidate label predicted by top1 is not suitable for the sentence intent, the correct answer can be accurately selected from the candidate labels.

6 is a structural diagram of a computer system according to an embodiment of the present invention.

Referring to FIG. 6 , a computer system 200 operated by at least one processor executes a program including instructions described for executing the operation of the present invention. The program may be stored in a computer readable storage medium and may be distributed. Here, the structure of the computer system 200 may be the structure of the automatic labeling device 100 according to an embodiment of the present invention.

The hardware of the computer system 200 may include at least one processor 210, memory 220, storage 230, and communication interface 240, and may be connected through a bus. In addition, hardware such as an input device and an output device may be included. The computer system 200 may be loaded with various software including an operating system capable of driving programs.

The processor 210 is a device for controlling the operation of the computer system 200, and may be various types of processors that process commands included in a program, for example, a Central Processing Unit (CPU) or a Micro Processor Unit (MPU). ), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), and the like.

Memory 220 loads a corresponding program so that the instructions described to carry out the operations of the present invention are processed by processor 210 . The memory 220 may be, for example, read only memory (ROM) or random access memory (RAM). The storage 230 stores various data, programs, etc. required to execute the operation of the present invention. The communication interface 240 may be a wired/wireless communication module.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims are also included in the scope of the present invention. that fall within the scope of the right.

Claims (9)

As a method for an automatic labeling device to label a spoken sentence,
Learning a classifier to receive learning data labeled with speech intention and output the speech intention from a learning sentence included in each of the training data;
generating a keyword phrasebook for the learned sentence by inputting the learning data into the learned classifier; and
Receiving a speech sentence whose speech intention is not labeled, and outputting the speech intention of the speech sentence using a plurality of candidate labels for the speech sentence using the learned classifier and a keyword phrasebook generated from the speech sentence step
including,
The classifier,
A language model learned to extract speech intent and keywords from the learning data and the speech sentence; and
An attention mechanism that receives the keyword extracted by the language model, extracts a plurality of words related to the keyword from the learning data and the spoken sentence, and generates a keyword phrasebook together with the keyword.
Including, labeling method. According to claim 1,
In the step of outputting the utterance intention,
comparing a first candidate label having the highest prediction reliability among a plurality of candidate labels generated from the utterance sentence with a preset threshold; and
If the prediction reliability of the first candidate label is greater than the threshold value, outputting the first candidate label as the correct answer of the spoken sentence and setting it as the correct answer label for the spoken sentence.
Including, labeling method. According to claim 2,
If the prediction reliability of the first candidate label is less than the threshold value, any one of the plurality of candidate labels excluding the first candidate label is set as the correct answer label based on the keyword phrasebook generated from the spoken sentence. steps to set up
Further comprising a labeling method. According to claim 3,
In the step of outputting the utterance intention,
extracting, by a self-attention mechanism included in the language model, a word for which a self-attention score is highest in the spoken sentence as a keyword of the spoken sentence;
extracting, by the attention mechanism, at least one related word related to the keyword extracted by the self-attention mechanism; and
generating the keyword phrasebook including the keyword and the at least one related word;
Including, labeling method. According to claim 4,
After the step of outputting the utterance intention,
Re-learning the classifier using the utterance sentence in which the correct answer label is set
Including, labeling method. An automatic labeling device for labeling an utterance intent in a utterance sentence,
A memory containing at least one instruction and storing labeled training data;
an interface that receives an unlabeled spoken sentence uttered by a speaker; and
processor
including,
the processor,
A learning sentence and an utterance intent for the learning sentence are extracted from the learning data, a classifier is trained using the learning sentence and an utterance intent, a keyword phrasebook for the learning sentence is created in the learned classifier, An automatic labeling device for inputting the spoken sentence into a classifier and labeling the spoken sentence with an intention for the spoken sentence. According to claim 6,
the processor,
The prediction reliability of speech intent is calculated from the speech sentence, a plurality of candidate labels are extracted, and a first candidate label having the highest prediction reliability among the extracted candidate labels is compared with a preset threshold, so that the first candidate label is obtained. An automatic labeling device that determines whether a correct label is correct. According to claim 7,
the processor,
If the prediction reliability of the first candidate label is greater than the threshold value, the first candidate label is output as a correct answer of the spoken sentence and set as a correct answer label for the spoken sentence;
If the prediction reliability of the first candidate label is less than the threshold value, any one of the plurality of candidate labels excluding the first candidate label is set as the correct answer label based on the keyword phrasebook generated from the spoken sentence. to set up, an automatic labeling device. According to claim 8,
the processor,
A word having the highest self-attention score calculated in the spoken sentence is extracted as a keyword of the spoken sentence, at least one related word related to the keyword is extracted, and the keyword including the keyword and the at least one related word is extracted. An automatic labeling device that generates phrasebooks.

Images