KR102195246B1 - Method of emotion recognition using audio signal, computer readable medium and apparatus for performing the method - Google Patents

Abstract

Disclosed are a method for classifying emotions using voice signals, and a recording medium and apparatus for performing the same. The emotion classification method using a voice signal includes the steps of constructing a plurality of emotion classification models obtained by learning a plurality of emotion-specific voice signals, and a plurality of emotion-specific emotion probabilities calculated from the user's voice signal using the plurality of emotion classification models. And determining an emotion appearing in the user's voice signal by acquiring and combining them.

Description

A method for classifying emotions using voice signals, a recording medium and a device for performing it {METHOD OF EMOTION RECOGNITION USING AUDIO SIGNAL, COMPUTER READABLE MEDIUM AND APPARATUS FOR PERFORMING THE METHOD}

The present invention relates to a method for classifying emotions using a voice signal, a recording medium and apparatus for performing the same, and more particularly, a method for classifying emotions using a voice signal for determining a person's emotions appearing in a voice signal, and a recording for performing the same It relates to media and devices.

With the development of data processing and cloud technology, the spread of services for single-person households is gaining popularity. Accordingly, the importance of communication between humans and machines is emerging so that human emotions can be judged and appropriate services are provided, and studies to judge human emotions using various signals such as faces, voices, and actions are actively conducted. It is being done.

For example, in the case of a voice signal, tremor, stress, and speech speed may vary according to a human's emotional state. Therefore, various services have been proposed that determine human emotions using such voice signals and use them. AT&T in the US is providing a service that recognizes and responds to customer's emotions in real time in a call center, and SoftBank in Japan is providing an emotion recognition robot that can recognize users' emotions using voice signals and communicate naturally.

Korean Patent Publication No. 10-2014-0007883 discloses an emotion recognition system using a voice signal. This classifies the user's emotions by extracting a specific parameter vector from the keyframe of the speech signal, and speech feature data such as a pre-learned Gaussian Mixture Model (GMM) or Hidden Markov Model (HMM). Since the user's emotions are classified based on the distribution model of, and feature parameter vectors are extracted only from a few selected keyframes, if the user's gender, vocal features, and speech phrases are different, the emotion discrimination performance may be degraded.

In addition, since the emotion classification method using voice signals has a problem that the accuracy is somewhat lower than that of the emotion classification method using facial expressions, research is required to supplement this.

One aspect of the present invention is a method for classifying emotions using a voice signal for discriminating emotions appearing in a voice signal using a plurality of machine learning classifiers using voice feature data that can be extracted from the voice signal as input data, and a recording for performing the same Provides media and devices.

The technical problem of the present invention is not limited to the technical problem mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In the emotion classification method using a voice signal of the present invention for solving the above problem, in the emotion classification method using a voice signal for classifying the user's emotion by receiving a user's voice signal, a plurality of emotions Constructing a plurality of emotion classification models from which voice signals are learned, and obtaining, for each of the plurality of emotion classification models, a plurality of emotion probabilities calculated from the voice signal of the user using the plurality of emotion classification models. , Computing an emotion determination confidence level for each of the plurality of emotions by combining the emotion probabilities of each of the plurality of emotions obtained for each of the plurality of emotion classification models, and the emotion determination confidence level for the plurality of emotions and the emotion determination for each of the plurality of emotions And determining an emotion appearing in the user's voice signal among the plurality of emotions by comparing the confidence level reference values.

On the other hand, the step of constructing a plurality of emotion classification models in which the plurality of emotion-specific speech signals are learned may include receiving a learning speech signal generated to represent any one of the plurality of emotions, from the learning speech signal. To extract speech feature data including at least one of pitch, zero crossing rate (ZCR), mel-scaled power spectrogram (MFCC), root mean square energy (RMSE), tempo, beat, and short-time fourier transform (STFT) And constructing the plurality of emotion classification models obtained by learning the at least one voice characteristic data for the emotion indicated by the learning voice signal.

In addition, the step of constructing a plurality of emotion classification models in which the plurality of emotion-specific speech signals are learned may include receiving a learning speech signal generated to represent any one of the plurality of emotions, wherein the learning speech signal is Labeling the learning speech signal in a binary form to identify whether one of the plurality of emotions represents an emotion, and inputting the labeled learning speech signal to a plurality of binary classification learning models, respectively Thus, it may include the step of constructing the plurality of emotion classification models.

In addition, the step of constructing the plurality of emotion classification models by inputting the labeled training speech signals into a plurality of binary classification learning models, respectively, comprises: applying the labeled training speech signals to a support vector machine model, an Xgboost model, and a random forest model. It may be a step of constructing the plurality of emotion classification models by inputting each.

In addition, the step of calculating an emotion discrimination confidence level for each of the plurality of emotions by combining the emotion probabilities of the plurality of emotions each obtained for each of the plurality of emotion classification models may include: Calculating a weight for each emotion classification model, and adding all values obtained by multiplying the weights calculated for each of the plurality of emotion classification models to the emotion probability obtained by each of the plurality of emotion classification models for any one emotion among the plurality of emotions. It may include calculating an emotion discrimination confidence level.

In addition, the step of calculating a weight for each of the plurality of emotion classification models for any one of the plurality of emotions includes determining the probability of each of the plurality of emotions in any one of the plurality of emotion classification models. It may include calculating the average and calculating a ratio of one emotion among the plurality of emotions to the average of the probability of each of the plurality of emotions as a weight of an emotion classification model for the corresponding emotion.

In addition, it may be a computer-readable recording medium in which a computer program is recorded for performing the emotion classification method using the voice signal.

On the other hand, the emotion classification apparatus using the voice signal of the present invention includes an emotion classification model construction unit for constructing a plurality of emotion classification models in which a plurality of emotion-specific voice signals are learned, and from the user's voice signal using the plurality of emotion classification models. An emotion probability calculation unit that obtains a calculated emotion probability for each of a plurality of emotions for each of the plurality of emotion classification models, and determines the emotion for each of the plurality of emotions by combining the emotion probability for each of the plurality of emotions obtained for each of the plurality of emotion classification models. An emotion determination confidence level calculation unit that calculates the confidence level and the emotion determination confidence level for each of the plurality of emotions are compared with a reference value of the emotion determination confidence level for each emotion to determine an emotion appearing in the user's voice signal among the plurality of emotions. It includes an emotion classification unit.

Meanwhile, the emotion probability calculation unit includes a pitch, a zero crossing rate (ZCR), a mel-scaled power spectrogram (MFCC), a root mean square energy (RMSE), a tempo, a beat, and a short-time fourier (STFT) from the voice signal of the user. transform), extracting voice feature data including at least one of the voice signals, and inputting them to a support vector machine model, an Xgboost model, and a random forest model, respectively, in which the at least one voice feature data is learned for the emotion represented by the voice signal. The emotion probability for each emotion may be obtained for each of the support vector machine model, the Xgboost model, and the random forest model.

Further, the emotion discrimination confidence degree calculation unit averages the probabilities of each of the plurality of emotions in any one of the plurality of emotion classification models, and the plurality of emotions with respect to the average of the probabilities of each of the plurality of emotions The ratio of one of the emotions is calculated as the weight of the corresponding emotion classification model for the corresponding emotion, and the plurality of emotion probabilities obtained for each of the plurality of emotion classification models for any one emotion among the plurality of emotions are calculated. The emotion discrimination confidence level can be calculated by adding all the values obtained by multiplying the weights calculated for each emotion classification model.

According to the present invention, by reflecting all of the voice feature data that can be extracted from the voice signal in the emotion classification using the voice signal, flexible emotion discrimination performance independent of the speaker and context can be exhibited.

In addition, it is possible to improve the accuracy of emotion classification by using a plurality of machine learning classifiers exhibiting high performance in binary classification and learning.

1 is a block diagram of an apparatus for classifying emotions using a voice signal according to an embodiment of the present invention.
2 to 7 are flowcharts of a method for classifying emotions using a voice signal according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION The detailed description of the present invention to be described below refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the present invention. It is to be understood that the various embodiments of the present invention are different from each other but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of the present invention, if appropriately described, is limited only by the appended claims, along with all scopes equivalent to those claimed by the claims. In the drawings, like reference numerals refer to the same or similar functions over several aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of an apparatus for classifying emotions using a voice signal according to an embodiment of the present invention.

Referring to FIG. 1, the emotion classification apparatus 1 using a voice signal according to an embodiment of the present invention includes an input unit 10, an emotion classification model construction unit 30, an emotion probability calculation unit 50, and an emotion discrimination confidence. It includes a degree calculation unit 70 and an emotion classification unit 90.

The apparatus 1 for classifying emotions using a voice signal according to an embodiment of the present invention may be implemented by more elements than those shown in FIG. 1, or may be implemented by fewer elements.

The emotion classification device 1 using a voice signal according to an embodiment of the present invention is a device capable of communication and input/output of information, and may be implemented by, for example, a smartphone, a tablet, a PC, etc., and the present invention Software (application) for classifying emotions may be installed and executed.

The configuration of the input unit 10, the emotion classification model construction unit 30, the emotion probability calculation unit 50, the emotion determination confidence level calculation unit 70, and the emotion classification unit 90 shown in FIG. 1 is one of the present invention. It can be controlled by software executed in the emotion classification apparatus 1 using a voice signal according to the embodiment.

The emotion classification apparatus 1 using a voice signal according to an embodiment of the present invention may determine a user's emotion from the user's utterance. The emotion classification apparatus 1 using a voice signal according to an embodiment of the present invention may determine a user's emotion appearing in the voice signal by analyzing a voice signal generated by a user's utterance.

The emotion classification apparatus 1 using a voice signal according to an embodiment of the present invention can exhibit flexible emotion discrimination performance by reflecting all feature data that can be extracted from the voice signal in the emotion classification using the voice signal. In addition, the emotion classification apparatus 1 using a speech signal according to an embodiment of the present invention inputs feature data extracted from the speech signal into a machine learning classifier and determines emotions appearing in the speech signal according to the output values. The accuracy of emotion discrimination can be improved by using a running classifier.

Hereinafter, each component of the emotion classification apparatus 1 using a voice signal according to an embodiment of the present invention shown in FIG. 1 will be described in detail.

The input unit 10 may receive a user's voice signal. For example, the input unit 10 may be implemented as a microphone to receive a voice signal generated by a user's speech. Alternatively, the input unit 10 may receive an audio signal previously stored in a computer-readable storage medium.

In this embodiment, the voice signal can be roughly divided into a learning voice signal and a test voice signal. The training speech signal is a speech signal used to build an emotion classification model, which is a machine learning classifier for emotion classification, and corresponds to a speech signal uttered by a user acting a specific emotion. The test voice signal corresponds to a voice signal subject to emotion classification. Hereinafter, for convenience of description, the voice signal is divided into a learning voice signal and a test voice signal.

In addition, in this embodiment, the emotion that appears in the voice signal is defined as the emotion of the user who generated the voice signal, and is regarded as an emotion consisting of one or more of anger, happiness, serenity, sadness, fear, disgust, and boredom. Here, the characteristics of the voice signal according to each emotion of anger, happiness, tranquility, sadness, fear, disgust, and boredom are generally shown in Table 1 below.

The emotion classification model construction unit 30 may build a plurality of emotion classification models obtained by learning a speech signal for training.

As described above, the emotion classification model is a machine learning classifier for emotion classification, and in this embodiment, a support vector machine model, an Xgboost model, and a random forest model, which have been proven to exhibit high performance in binary classification, may be adopted.

The emotion classification model construction unit 30 may construct a total of three emotion classification models by inputting a training speech signal to a support vector machine model, an Xgboost model, and a random forest model, respectively.

Here, the learning voice signal may be divided into a plurality of emotions as described above. For example, the learning voice signal may be a voice signal in which any one of anger, happiness, tranquility, sadness, fear, disgust, and boredom appears, or may be a voice signal uttered in a normal emotional state in which the above-described emotion does not appear.

The emotion classification model building unit 30 inputs the plurality of emotion-specific learning speech signals to a support vector machine model, an Xgboost model, and a random forest model, respectively, and builds a plurality of emotion classification models in which speech signals for each emotion are learned. can do.

To this end, the emotion classification model building unit 30 extracts at least one voice feature data representing a feature of the voice signal from the training voice signal, and identifies whether the training voice signal represents any one of a plurality of emotions. To do this, it can be labeled in the form of binary.

Specifically, the emotion classification model building unit 30 may separate the training speech signal into unit frames having a predetermined size. For example, the emotion classification model building unit 30 may separate the training speech signal into a plurality of unit frames having a size of 20 to 30 ms.

The emotion classification model building unit 30 includes a pitch, a zero crossing rate (ZCR), a mel-scaled power spectrogram (MFCC), a root mean square energy (RMSE), a tempo, a beat, and a short-time fourier (STFT) for each of a plurality of unit frames. transform) can be extracted. Pitch, ZCR, MFCC, RMSE, tempo, beat, and STFT are representative features that can be extracted from a voice signal, and their definitions are shown in Table 2 below.

The emotion classification model construction unit 30 may level at least one voice feature data extracted from each of a plurality of unit frames as a global feature. For example, the emotion classification model building unit 30 equalizes at least one speech feature data each extracted from a plurality of unit frames in any one of a maximum value, a minimum value, an average value, and a standard deviation value to display the training speech data as a whole. You can do it.

The emotion classification model construction unit 30 constructs a plurality of emotion classification models by inputting at least one speech feature data extracted from the learning speech data and leveled in this way into a plurality of emotion classification models, respectively. At least one voice feature data may be labeled in a binary form so as to identify whether the voice signal for poetry learning represents any one of a plurality of emotions.

For example, the emotion classification model building unit 30 may first build a plurality of emotion classification models in which voice feature data for anger among a plurality of emotions is learned. At this time, the emotion classification model building unit 30, when the speech signal for learning is a speech signal that delays anger, labels at least one speech feature data extracted from the speech signal for learning as '1' so that the speech signal for learning is true. In the case of a voice signal that utters in a normal emotional state or postpones another emotion, at least one voice feature data extracted from the corresponding learning voice signal may be labeled as '0' so that it becomes false.

The emotion classification model construction unit 30 may input the voice feature data labeled as described above into a plurality of emotion classification models, respectively, to construct a plurality of emotion classification models in which voice feature data for each emotion is learned.

The emotion probability calculation unit 50 may obtain emotion probabilities for a plurality of emotions by respectively inputting test voice signals to a plurality of emotion classification models. In this embodiment, the emotion probability means the probability that a specific emotion appears in the voice signal. The emotion probability calculator 50 may obtain an emotion probability for each emotion for each of a plurality of emotion classification models for one test voice signal.

To this end, the emotion probability calculation unit 50 may extract at least one voice feature data representing a feature of the voice signal from the test voice signal.

Specifically, like the emotion classification model building unit 30, the emotion probability calculation unit 50 divides the test voice signal into unit frames of a predetermined size, and the pitch, ZCR (Zero Crossing Rate), and Speech feature data including at least one of mel-scaled power spectrogram (MFCC), root mean square energy (RMSE), tempo, beat, and short-time fourier transform (STFT) is extracted, and at least each extracted from a plurality of unit frames One voice feature data can be equalized to a global feature so that the test voice data can be displayed as a whole.

The emotion probability calculation unit 50 may obtain an emotion probability for each emotion for the test speech signal by inputting at least one speech feature data extracted and leveled from the test speech data into a plurality of emotion classification models, respectively. . The emotion classification model based on the support vector machine model, Xgboost model, and random forest model, when the voice feature data extracted from the test voice signal is input, the input voice feature data and learned anger, happiness, tranquility, sadness, fear, disgust. , By calculating the similarity of voice characteristic data of seven emotions of boredom, emotion probabilities indicating the probability of each emotion of anger, happiness, tranquility, sadness, fear, disgust, and boredom appearing can be output for the test voice signal.

The emotion determination confidence level calculation unit 70 may calculate the emotion determination confidence level for each emotion by combining emotion probabilities for each emotion obtained for each of a plurality of emotion classification models with respect to the test voice data. In this embodiment, the confidence level of emotion determination is a value used as a criterion for determining emotions appearing in a voice signal, and may have a value between 0 and 1.

The emotion discrimination confidence level calculation unit 70 may calculate weights for each of a plurality of emotion classification models to be reflected in a combination of emotion probabilities for each emotion obtained for each of a plurality of emotion classification models for test voice data as shown in Equation 1 below. have.

For example, in the support vector machine model, the emotion discrimination confidence level calculation unit 70 may calculate an average of the plurality of emotion probabilities calculated for the test voice signal from the support vector machine model. That is, the emotion discrimination confidence level calculation unit 70 may calculate an average of the emotional probabilities of anger, happiness, tranquility, sadness, fear, disgust, and boredom calculated from the support vector machine model for the test voice signal. The emotion discrimination confidence level calculation unit 70 may calculate a ratio of a specific emotion probability to an average of a plurality of emotion probabilities as a weight of an emotion classification model for a corresponding emotion. That is, the emotion discrimination confidence level calculation unit 70 may calculate the weight of the anger emotion of the support vector machine model as a ratio of the anger emotion probability to the average of the total emotion probability.

As shown in Equation 2 below, the emotion discrimination confidence calculation unit 70 multiplies the corresponding emotion probability obtained by each of the plurality of emotion classification models by the weight calculated for each of the plurality of emotion classification models, as shown in Equation 2 below. By adding all the values, you can calculate the confidence level of emotion discrimination.

For example, when the emotion discrimination confidence level calculation unit 70 calculates the emotion discrimination confidence level for the anger emotion that appears in the test voice signal, the weight for the anger emotion of the support vector machine model and the test voice signal A value multiplied by the anger emotion probability calculated from the support vector machine model, the weight for the anger emotion in the Xgboost model, multiplied by the anger emotion probability calculated from the Xgboost model for the test voice signal, and the weight for the anger emotion in the random forest model. And by adding all the values obtained by multiplying the test voice signal by the anger emotion probability calculated from the random forest model, the confidence level of the emotion discrimination for the anger emotion of the test voice signal can be calculated.

In this way, the emotion determination confidence level calculation unit 70 may calculate the emotion determination confidence level of each emotion of anger, happiness, tranquility, sadness, fear, disgust, and boredom with respect to the test voice signal. According to the present embodiment, a speech signal is generated by calculating an emotion discrimination confidence level used for classifying emotions appearing in a speech signal by combining emotion probabilities calculated from three emotion classification models of a support vector machine model, an Xgboost model, and a random forest model. The accuracy of the used emotion classification can be improved.

The emotion classifying unit 90 may determine at least one emotion appearing in the test voice signal among the plurality of emotions by using the confidence level of determination of emotions for each of the plurality of emotions with respect to the test voice signal.

The emotion classifying unit 90 may compare the emotion determination confidence level for each emotion with a plurality of emotion determination confidence level reference values for the test voice signal. In the present embodiment, the emotion discrimination confidence reference value for a plurality of emotions may all be set equally to 0.5, and according to another embodiment, it may be set differently for a plurality of emotions.

For example, the emotion classifying unit 90 may determine that the anger emotion appears in the test voice signal when the emotion discrimination confidence level of the anger emotion with respect to the test voice signal is 0.5 or more.

In this way, the emotion classification apparatus 1 using a voice signal according to an embodiment of the present invention reflects all voice characteristic data that can be extracted from the voice signal in the emotion classification using the voice signal, so that it is independent of the speaker and the context. It can show flexible emotion discrimination performance, and the accuracy of emotion classification can be improved by using a plurality of machine learning classifiers that have high performance in binary classification and learning.

Hereinafter, a method for classifying emotions using a voice signal according to an embodiment of the present invention will be described.

The emotion classification method using a voice signal according to an embodiment of the present invention may be performed in substantially the same configuration as the emotion classification apparatus 1 using a voice signal according to the embodiment of the present invention shown in FIG. 1. Accordingly, the same components as those of the emotion classification apparatus 1 using the voice signal of FIG. 1 are given the same reference numerals, and repeated descriptions are omitted.

2 to 7 are flowcharts of a method for classifying emotions using a voice signal according to an embodiment of the present invention.

The emotion classification method using the voice signal of the present invention can be largely divided into a step of constructing an emotion classification model for classifying emotions and a step of classifying emotions using the emotion classification model. Hereinafter, a step of constructing an emotion classification model will be described with reference to FIGS. 2 to 4, and an emotion classification step will be described with reference to FIGS. 5 to 7.

First, referring to FIG. 2, the input unit 10 may receive an audio signal (S100).

Here, the speech signal is a speech signal used to build an emotion classification model, which is a machine learning classifier for emotion classification, and corresponds to a learning speech signal uttered while a user plays a specific emotion.

The emotion classification model building unit 30 may build an emotion classification model obtained by learning a voice signal (S200).

The emotion classification model construction unit 30 may perform a predetermined preprocessing step on the speech signal input from the input unit 10 in order to learn the speech signal in the emotion classification model. This will be described with reference to FIG. 3.

Referring to FIG. 3, the emotion classification model construction unit 30 includes a pitch, a zero crossing rate (ZCR), a mel-scaled power spectrogram (MFCC), a root mean square energy (RMSE), a tempo, a beat, and a STFT ( Voice feature data including at least one of short-time fourier transform) may be extracted (S110).

The emotion classification model construction unit 30 separates the speech signal into unit frames of a predetermined size, and extracts speech feature data including at least one of pitch, ZCR, MFCC, RMSE, tempo, beat, and STFT for each of a plurality of unit frames. In addition, at least one voice feature data extracted from each of a plurality of unit frames may be equalized as a global feature.

When the voice signal represents emotion (S120), the emotion classification model construction unit 30 binarizes the voice characteristic data to be true for the emotion represented by the voice signal (S130), and when the voice signal does not represent emotion (S120). ), the voice characteristic data may be binarized so as to be false with respect to the emotion indicated by the voice signal (S140).

The emotion classification model building unit 30 may label at least one voice feature data extracted from voice data and leveled in a binary form. For example, when the voice signal is a voice signal that postpones anger, the emotion classification model construction unit 30 labels at least one voice characteristic data extracted from the corresponding voice signal as '1' so that the voice signal is true. In the case of a voice signal that utters in a normal emotional state or postpones another emotion, at least one voice feature data extracted from the corresponding voice signal may be labeled as '0' so that it is false.

Referring to FIG. 4, the emotion classification model building unit 30 inputs voice feature data binarized with respect to the emotion indicated by the voice signal to the support vector machine model, the Xgboost model, and the random forest model (S210), and the voice signal is A plurality of emotion classification models for each displayed emotion may be constructed (S220).

As described above, the emotion classification model is a machine learning classifier for emotion classification, and in this embodiment, a support vector machine model, an Xgboost model, and a random forest model, which have been proven to exhibit high performance in binary classification, may be adopted.

The emotion classification model building unit 30 may construct a total of three emotion classification models by inputting a speech signal to a support vector machine model, an Xgboost model, and a random forest model, respectively.

The emotion classification model building unit 30 inputs a plurality of voice signals for each emotion into a support vector machine model, an Xgboost model, and a random forest model, respectively, and builds a plurality of emotion classification models in which the plurality of emotion-specific voice signals are learned. I can.

Referring to FIG. 5, the input unit 10 may receive an audio signal (S300).

Here, the voice signal corresponds to a test voice signal that is subject to emotion classification.

The emotion probability calculation unit 50 may input the voice signals to the plurality of emotion classification models (S400), and calculate emotion probabilities for each of the plurality of emotions for the voice signals from the plurality of emotion classification models (S500).

The emotion probability calculation unit 50 may obtain emotion probabilities for a plurality of emotions by respectively inputting a voice signal to a plurality of emotion classification models. In this embodiment, the emotion probability means the probability that a specific emotion appears in the voice signal. The emotion probability calculation unit 50 may obtain an emotion probability for each emotion for each of a plurality of emotion classification models for one voice signal.

The emotion probability calculation unit 50 divides the speech signal into unit frames of a certain size, and a pitch, a zero crossing rate (ZCR), a mel-scaled power spectrogram (MFCC), and a root mean square energy (RMSE) for each of a plurality of unit frames , Tempo, tempo, and short-time fourier transform (STFT), extracting voice feature data including at least one of, and leveling at least one voice feature data extracted from each of a plurality of unit frames into a global feature to represent the voice data as a whole. You can do it.

The emotion probability calculation unit 50 may obtain an emotion probability for each emotion with respect to the test speech signal by inputting at least one speech feature data extracted from the speech data and leveled into the plurality of emotion classification models, respectively. The emotion classification model based on the support vector machine model, Xgboost model, and random forest model, when the voice feature data extracted from the voice signal is input, the input voice feature data and the learned anger, happiness, tranquility, sadness, fear, disgust, boredom By calculating the similarity of the voice characteristic data of the seven emotions of, the emotion probabilities indicating the probability of each emotion appearing in the voice signal can be outputted for anger, happiness, tranquility, sadness, fear, disgust, and boredom.

The emotion determination confidence level calculation unit 70 may calculate the emotion determination confidence level for each emotion by combining the emotion probabilities of a plurality of emotions with respect to the voice signal (S600). In this regard, a detailed description will be described later with reference to FIG. 6.

The emotion classifying unit 90 may determine an emotion appearing in the voice signal by using the emotion discrimination confidence level for each emotion (S700). A detailed description in this regard will be described later with reference to FIG. 7.

Referring to FIG. 6, the emotion determination confidence level calculation unit 70 may calculate weights for each of a plurality of emotions of each of the plurality of emotion classification models (S610).

The emotion discrimination confidence level calculation unit 70 may calculate weights for seven emotions of anger, happiness, tranquility, sadness, fear, disgust, and boredom of each of the plurality of emotion classification models according to Equation 1.

The emotion determination confidence level calculation unit 70 may calculate the emotion determination confidence level for a plurality of emotions by using the emotion probabilities for each emotion and the weights for each emotion of each of the plurality of emotion classification models (S620).

The emotion discrimination confidence level calculation unit 70 is a value obtained by multiplying the corresponding emotion probability obtained by each of the plurality of emotion classification models for any one emotion among the plurality of emotions by the weight calculated for each of the plurality of emotion classification models, as shown in Equation 2 You can calculate the confidence level of emotion discrimination by adding all of them.

For example, when the emotion discrimination confidence level calculation unit 70 calculates the emotion discrimination confidence level for the anger emotion appearing in the voice signal, the weight for the anger emotion of the support vector machine model and the support vector for the test voice signal Multiplied by the anger emotion probability calculated from the machine model, the weight for the anger emotion in the Xgboost model and the product of the anger emotion probability calculated from the Xgboost model for the test voice signal, and the weight and test for the anger emotion in the random forest model. The emotion discrimination confidence level for the anger emotion of the speech signal can be calculated by adding all the values obtained by multiplying the anger emotion probability calculated from the random forest model with respect to the dragon speech signal.

Referring to FIG. 7, the emotion classifying unit 90 determines the emotion as an emotion appearing in a voice signal when the confidence level of determination of emotion for a plurality of emotions is 0.5 or more (S710), and determines emotion for a plurality of emotions When the confidence level is less than 0.5 (S710), the corresponding emotion may be determined as an emotion that does not appear in the voice signal (S730).

The emotion classifying unit 90 may compare the emotion determination confidence level for each emotion of the voice signal and a reference value of the emotion determination confidence level for each emotion. In the present embodiment, the reference value of the emotion discrimination confidence level for a plurality of emotions may all be set equally to 0.5, and according to another embodiment, it may be set differently for a plurality of emotions.

For example, the emotion classifying unit 90 may determine that the anger emotion appears in the voice signal when the emotion discrimination confidence level of the anger emotion relative to the voice signal is 0.5 or more.

As described above, the emotion classification method using the voice signal of the present invention may be implemented as an application or implemented in the form of program instructions that may be executed through various computer components, and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded in the computer-readable recording medium may be specially designed and constructed for the present invention, and may be known and usable to those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magnetic-optical media such as floptical disks. media), and a hardware device specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

Although the above has been described with reference to embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention described in the following claims. I will be able to.

1: emotion classification device using voice signals
10: input
30: emotion classification model construction unit
50: emotion probability calculation unit
70: emotion discrimination confidence level calculation unit
90: emotion classification unit

Claims (10)

In the emotion classification method in an emotion classification apparatus using a voice signal for receiving a user's voice signal and classifying the user's emotion,
Building a plurality of emotion classification models in which voice signals for each emotion are learned;
Obtaining, for each of the plurality of emotion classification models, a plurality of emotion probabilities calculated from the voice signal of the user using the plurality of emotion classification models;
Calculating an emotion discrimination confidence level for each of the plurality of emotions by combining the emotion probabilities for each of the plurality of emotions, respectively acquired for each of a plurality of emotion classification models; And
Comprising the step of comparing the emotion determination confidence level of the plurality of emotions and the emotion determination confidence level of the plurality of emotions to determine an emotion appearing in the voice signal of the user among the plurality of emotions,
Computing an emotion discrimination confidence level for each of the plurality of emotions by combining the emotion probabilities of the plurality of emotions each obtained for each of the plurality of emotion classification models,
Calculating a weight for each of the plurality of emotion classification models for any one of the plurality of emotions; And
Computing an emotion discrimination confidence level by adding all values obtained by multiplying the weights calculated for each of the plurality of emotion classification models to the emotion probabilities obtained for each of the plurality of emotion classification models for any one of the plurality of emotions. And
The step of calculating a weight for each of the plurality of emotion classification models for any one of the plurality of emotions,
Averaging the probability of each of the plurality of emotions in one of the plurality of emotion classification models; And
And calculating a ratio of an emotion probability of any one of the plurality of emotions to an average of the probabilities of each of the plurality of emotions as a weight of an emotion classification model for the corresponding emotion,
The step of determining an emotion appearing in the voice signal of the user among the plurality of emotions by comparing the emotion determination confidence level for each emotion and the emotion determination confidence level for the plurality of emotions,
When the emotion discrimination confidence level for any one of the plurality of emotions is equal to or greater than the reference value for the emotion discrimination confidence level for the corresponding emotion, the emotion is determined as an emotion appearing in the voice signal of the user, and the emotion for each of the plurality of emotions An emotion classification method using a voice signal in which the discrimination confidence reference value is set differently for each of the plurality of emotions. The method of claim 1,
Building a plurality of emotion classification models in which the plurality of emotion-specific voice signals are learned,
Receiving a speech signal for learning generated to represent one of the plurality of emotions;
Voice including at least one of pitch, zero crossing rate (ZCR), mel-scaled power spectrogram (MFCC), root mean square energy (RMSE), tempo, beat, and short-time fourier transform (STFT) from the training speech signal Extracting feature data; And
And constructing the plurality of emotion classification models obtained by learning the at least one voice characteristic data for the emotion indicated by the training voice signal. The method of claim 1,
Building a plurality of emotion classification models in which the plurality of emotion-specific voice signals are learned,
Receiving a speech signal for learning generated to represent one of the plurality of emotions;
Labeling the learning speech signal in a binary form so as to identify whether the learning speech signal represents any one of the plurality of emotions; And
And constructing the plurality of emotion classification models by inputting the labeled speech signals for learning into a plurality of binary classification learning models, respectively. The method of claim 3,
The step of constructing the plurality of emotion classification models by respectively inputting the labeled speech signals for training into a plurality of binary classification learning models,
An emotion classification method using a speech signal, comprising inputting the labeled speech signal for training into a support vector machine model, an Xgboost model, and a random forest model, respectively, to construct the plurality of emotion classification models. delete delete A computer-readable recording medium having a computer program recorded thereon for performing the method for classifying emotions using the voice signal according to claim 1. An emotion classification model construction unit for constructing a plurality of emotion classification models obtained by learning voice signals for each emotion;
An emotion probability calculator configured to obtain, for each of the plurality of emotion classification models, a plurality of emotion probabilities calculated from a voice signal of a user using the plurality of emotion classification models;
An emotion discrimination confidence level calculating unit for calculating an emotion discrimination confidence level for each of the plurality of emotions by combining the emotion probabilities for each of the plurality of emotions obtained for each of a plurality of emotion classification models; And
And an emotion classifying unit configured to determine an emotion appearing in the voice signal of the user among the plurality of emotions by comparing the emotion determination confidence level for each emotion and a reference value for emotion determination confidence level for the plurality of emotions,
The emotion discrimination confidence level calculation unit calculates a weight for each of the plurality of emotion classification models for any one of the plurality of emotions, and for each of the plurality of emotion classification models for any one of the plurality of emotions The emotion discrimination confidence level is calculated by adding all the values obtained by multiplying the obtained emotion probability by the weight calculated for each of the plurality of emotion classification models, and in one emotion classification model among the plurality of emotion classification models, each of the plurality of emotions Probability is averaged, and a ratio of an emotion probability of any one of the plurality of emotions to the average of the probability of each of the plurality of emotions is calculated as a weight of an emotion classification model for the emotion,
The emotion classification unit, when the emotion determination confidence level for any one emotion among the plurality of emotions is equal to or greater than a reference value for the emotion determination confidence level for the corresponding emotion, determines the emotion as an emotion appearing in the voice signal of the user, An emotion classification apparatus using a voice signal in which a plurality of emotion discrimination confidence reference values are set differently for each of the plurality of emotions. The method of claim 8,
The emotion probability calculation unit,
Including at least one of a pitch, a zero crossing rate (ZCR), a mel-scaled power spectrogram (MFCC), a root mean square energy (RMSE), a tempo, a beat, and a short-time fourier transform (STFT) from the user's voice signal. The voice feature data is extracted, and the at least one voice feature data is learned for the emotion represented by the voice signal and input into a support vector machine model, an Xgboost model, and a random forest model, respectively, and the emotion probabilities for a plurality of emotions are calculated as the support vector A device for classifying emotions using voice signals acquired for each machine model, Xgboost model, and random forest model. delete

Images