KR20040038419A - A method and apparatus for recognizing emotion from a speech - Google Patents

Abstract

The present invention relates to an emotion recognition system and an emotion recognition method using voice, and more particularly, to extract a voice for effectively estimating a human emotion state by extracting and analyzing various parameters representing a human emotion state from a spoken voice signal. It relates to an emotion recognition system and an emotion recognition method used.

The emotion recognition method using the voice according to the present invention includes a speech input step of receiving a speech signal according to a speaker's speech, a non-verbal parameter extraction step of extracting at least one non-verbal parameter from the speech signal, and the speech signal. Linguistic parameter extraction step of extracting at least one linguistic parameter according to the meaning of the spoken sound by speech recognition; And an emotional state calculation step of finally calculating the emotional state of the speaker by combining the linguistic parameters and the non-verbal parameters.

Description

Emotion Recognition System and Emotion Recognition Method Using Voice {A METHOD AND APPARATUS FOR RECOGNIZING EMOTION FROM A SPEECH}

The present invention relates to an emotion recognition system and an emotion recognition method using voice, and more particularly, to extract a voice for effectively estimating a human emotional state by extracting and analyzing various parameters representing a human emotional state from a spoken voice signal. It relates to an emotion recognition system and an emotion recognition method used.

Voice is the most meaningful means of human communication, a means of information delivery, and a means of language.

Attempts to implement communication through voice between humans and machines have been steadily evolving in the past. Moreover, the recent field of speech information technology (SIT) for effectively processing voice information has made remarkable progress. As a result, it is being applied to real life one after another.

If the voice information processing technology is largely classified, it may be classified into categories such as speech recognition, speech synthesis, speaker identification and verification, and speech coding.

Among these, speech recognition is a technique for recognizing spoken speech and converting it into a string, and speech synthesis is a technique for converting a string into original speech using data or parameters obtained from speech analysis. It is a technique for estimating or authenticating a talker through speech coding, and speech coding is a technique for effectively compressing and encoding a speech signal.

On the other hand, with respect to the speech recognition technology, it is possible to think of an emotion recognition technology that estimates and recognizes the speaker's emotional state through the speech signal.

Emotion recognition technology is ultimately aimed at realizing the interface between man and machine as he talks through the language, voice, and gestures that people use in their daily lives. It can be divided into emotion recognition in the auditory aspect that recognizes emotion and emotion recognition in the visual aspect that recognizes emotion through the speaker's expression. The present invention relates to emotional recognition in terms of hearing.

In general, a person can estimate the emotional state to some extent only by hearing another person's voice. According to Fukuda's research, humans can estimate the emotional state to some extent only through the non-verbal information of the speaker's speech itself even when no language information is given (S. Fukuda and V. Kostov, "Extracting"). Emotion from Voice, "Proc. Of IEEE, pp. IV-299-304, 1999.).

For example, the utterance of an angry and angry speaker not only increases the average pitch higher than in a normal state, but also changes the average pitch in a sentence even if a sentence consisting of the same word is spoken. have. In addition, since the voice becomes louder, the energy is increased relatively as compared to the case of the normal state in the same environment, and the average energy change is also relatively rapidly changed. Fukuda extracts pitch information using the cepstrum method, one of the representative methods of speech analysis, and then calculates the average pitch value obtained from the voice data collected in the normal state and the average pitch value obtained in the angry state. The emotional state of the spoken voice was extracted and experiments were performed.

In addition, G. Zhou also used the average pitch value of the speech to recognize the emotional state of the voice.However, unlike Fukuda, a teaser energy operator (TEO), which is superior to the cepstrum in estimating the pitch, And G. Zhou, JHL Hansen, and JF Kaiser, "Classification of Speech under Stress based on Features derived from the Nonlinear Teager Energy Operator , "Proc. Of ICASSP, pp. 549-552, 1998.).

The pitch range is generally in the range of 60Hz to 750Hz, and when the average value of the pitch is called AvgP, the AvgP in anger with the same person shows a relatively higher phenomenon than the AvgP in the normal state. In this way, the average value of the pitch can be used as a reference for recognizing the emotional state of the speaker.

Similarly, speech energy, a parameter that represents the loudness of speech, also plays a very important role in emotion extraction. For example, speech in anger with the same person increases in energy more than speech in a normal state because the energy of sound waves is proportional to the square of its amplitude. Therefore, in a state of anger, the voice has a high tendency to yell, so in general, when the voice energy is measured to be significantly larger than the general case, it can be estimated that the emotional state of the speaker is anger state.

There are many areas that can contribute due to the provision of a reliable emotion recognition system. For example, in the operation of a call center that requires direct customer service, if the customer's feelings are furious, the ARS autoresponder will be connected directly to the agent or even if the agent is connected. It is possible to build a system that will lead to more experienced and experienced counselors. In addition, by building a database of vocabulary that a customer speaks in a certain emotional state, it can be used as a material for future Customer Relation Management (CRM). In addition, by learning from the data classified according to the speaker's emotional state, the optimal speech recognition system can be implemented to improve the overall speech recognition rate of the speech recognition system. By not doing so, the dissatisfaction with the voice recognition service itself can be eliminated, thereby improving service satisfaction.

As such, the system that recognizes emotions through the speaker's speech has a wide range of useful applications. Since Fukuda's research, many technologies related to the speaker's emotion recognition through the speech have been developed and applied worldwide. In the prior art, most of the prior art recognizes the emotion of the speaker by using parameters related to the pitch average value or the energy level, as in the contents of Fukuda's research.

However, there are some problems in recognizing the emotion of the speaker by using the parameter of the average value or energy level of the pitch. For example, the average value of pitch is very large depending on the speaker, and even in the case of voice energy, external environmental factors such as the volume of the microphone, the state of the telephone line, and the surrounding conditions are very sensitive to the performance of emotion recognition. will be.

Therefore, in order to recognize the emotion state of the speaker more accurately, in addition to the parameters included in the speech used in the prior art, new parameters reflecting the speaker's emotion state should be extracted, and the data measured by these parameters should be taken into consideration comprehensively.

Accordingly, the present inventors propose a plurality of new parameters in addition to the parameters used in the emotion recognition system according to the prior art, and provide an emotion recognition system that appropriately aggregates the data measured by these parameters.

For example, focusing on the fact that the average value of pitches varies greatly from person to person, consider the average pitch defference of the pitch, and minimize the interference caused by external factors in the surrounding environment when measuring the energy of the pitch. For this purpose, consider the average energy difference. Also, in the case of a speaker who is in an emotional state, the speech rate is very fast, and the average speech rate is recognized. One of the factors of judgment is to

Furthermore, in the prior art emotion recognition system, it is noted that the emotional state of the speaker can be estimated through the non-verbal information of the spoken sounds, but there is a problem of overlooking the linguistic information, that is, the grasp of the contents of the speech. . For example, a speaker (eg, Gyeongsangdo dialect speaker, etc.) who speaks quickly in a furious tone may be mistaken for an angry state in spite of the content of speech, even though the emotion is not furious. Is there. In addition, vice versa. Applicant intends to provide a more reliable emotion recognition system by using not only non-verbal information but also linguistic information as a reference for emotion recognition.

The present invention is an emotion recognition system using speech that comprehensively extracts and analyzes a plurality of parameters related to linguistic and nonverbal information indicating a speaker's emotional state in order to effectively estimate and recognize the speaker's emotional state. And an emotion recognition method.

1 is a flowchart showing an example of a emotion recognition method using voice according to the present invention;

2 is a flow chart illustrating the flow chart of FIG. 1 in more detail;

3 is a flowchart illustrating a process of calculating an emotional state of a speaker in an emotional state calculating step by combining measured values of each parameter in an example of an emotion recognition method using voice according to the present invention;

4 is a configuration diagram showing an example of an emotion recognition system using voice according to the present invention;

5 is a block diagram illustrating an example of a call center counselor connection system using an emotion recognition system using voice according to the present invention.

In order to achieve the above object, the emotion recognition method using the voice according to the present invention, a speech input step for receiving a speech signal according to the speaker's speech, and non-verbal parameters for extracting at least one non-verbal parameter from the speech An extracting step, a linguistic parameter extraction step of extracting at least one linguistic parameter according to the meaning of the spoken sound by speech recognition of the speech signal, and the emotional state of the speaker by combining the linguistic parameters and the nonverbal parameters It includes a step of calculating the emotional state to the final calculation.

In this case, the non-verbal parameter is obtained by dividing the speech signal into speech signal frames at predetermined time intervals, and dividing the sum of the pitches of the voiced frames among the frames by the number of the voiced frames. It is preferable to include an average value.

The non-verbal parameter further includes a variance of the pitch of the speech signal obtained by dividing the sum of the squares of the difference between the mean value of the pitch and the pitch in each of the voiced frames by the number of frames of the voiced sounds. It is preferable to include.

In addition, the non-verbal parameter is the speech energy of the speech signal obtained by dividing the speech signal into speech signal frames at predetermined time intervals, and dividing the sum of the speech energy in the frames that are voiced sounds by the number of frames that are voiced sounds. It is preferable to include the average value of.

Further, the non-verbal parameter is a dispersion of speech energy of the speech signal obtained by dividing the sum of the squares of the difference between the mean value of the speech energy and the speech energy in each of the voiced frames by the number of frames of the voiced sound. variance).

In addition, the linguistic parameter preferably includes the speech rate of the voice signal.

At this time, the speech rate of the speech signal may be a relative speed of an average value of actual phoneme lengths of the speech signal to an average value of standard phoneme lengths of the speech signal. In addition, the average value of the standard phoneme length of the speech signal is each Korean phonetic code constituting the speech signal recognized by the speech recognition module, among the phonetic phoneme already stored in the unit sound database of the speech recognition module. It can be set as the average value of phoneme length.

In addition, the average value of the actual phoneme length of the speech signal, the sum of the length of each phonetic phoneme constituting the speech signal multiplied by the weight of the phonetic phoneme phonemes, the number of the number of phonetic phoneme stored in the speech recognition module Can be divided by.

At this time, it is preferable that the weight of the neutral phonetic phoneme of the phonetic phoneme is higher than the weight of the initial and the finality of the phonetic phoneme.

In addition, the linguistic parameter may include an emotion calculation coefficient corresponding to each special word included in a result of recognizing the voice signal by the voice recognition module.

The emotion state calculating step may include: a difference obtained by dividing a difference between an actual pitch average value and a standard pitch average value by the first weight multiplied by a value obtained by dividing the difference by the standard pitch average value, and a difference between an average value of actual voice energy and an average value of standard voice energy. Multiplied by the average value of the standard speech energy multiplied by the second weight, and the difference between the standard deviation of the actual pitch and the standard deviation of the standard pitch divided by the standard deviation of the mean standard pitch, multiplied by the third weight. A value obtained by dividing the difference between the standard deviation of the actual voice energy and the standard deviation of the standard voice energy by the standard deviation of the standard voice energy by the fourth weight, and the average value of the actual phoneme length and the average phoneme length. The difference between and is divided by the average value of the reference phoneme length multiplied by the fifth weight, and corresponding to the special words included in the result of speech recognition The final emotional state coefficient that is calculated by the sum of a value obtained by multiplying a sixth weight to the emotion calculation coefficient, it is preferable to calculate the emotional state of the speaker by comparing with a predetermined threshold value.

In addition, the emotion recognition system using the voice according to the present invention comprises a voice signal input unit, a signal processing unit, a voice recognition unit, a control unit and an output unit, the voice signal input unit, the electrical signal of the speaker voice Is converted to and transmitted to at least one of the signal processing unit and the speech recognition unit, and the signal processing unit includes: a pitch average value extracting unit extracting an average value of the pitch of the speech signal, and a pitch distribution extracting a degree of change of the average value of the pitch; and a variance extracting unit, a voice energy average value extracting unit extracting an average value of voice energy of the voice signal, and a voice energy dispersion extracting unit extracting a degree of change of the average value of the voice energy. A speech rate extracting unit for extracting a speech rate of the speech signal and a specific word included in the speech signal; Further includes a speech recognition result extracting unit, wherein the controller is configured to calculate the pitch average value, the pitch variance, the speech energy average value, the speech energy dispersion, the speech rate, and the speech recognition result calculated by the signal processor and the speech recognition unit. The final emotion recognition coefficient is calculated using at least one of the parameters, and the emotion state of the speaker is recognized by comparing the final emotion recognition coefficient with a predetermined threshold value.

In addition, the call center agent connection system using the emotion recognition system according to the present invention, the private branch exchange connected to the customer's telephone line, the CTI server for controlling the overall operation of the call center, and the emotion recognition server with a built-in emotion recognition system And an answering machine connected to the CTI server and at least one telephone line for counseling connected to the CTI server, wherein the emotion recognition server recognizes the emotion state of the customer from the voice signal according to the customer's speech. And, according to the result, the CTI server switches the call connection of the customer's telephone line from the answering machine to the agent's telephone line, or connects a call from one agent's telephone line to another telephone line. Characterized in that to switch.

Hereinafter, with reference to the accompanying drawings will be described in more detail.

1 is a flowchart illustrating an example of a emotion recognition method using voice according to the present invention.

As described above, the emotion recognition method using the voice according to the present invention intends to provide a more reliable emotion recognition system by using not only non-verbal information but also verbal information as a reference for emotion recognition.

As shown in FIG. 1, according to the emotion recognition method using the voice according to the present invention, first, the speaker's speech is input by an appropriate voice input means (microphone, etc.) in the speech input step 100, and then non-verbal. In the parameter extraction step 110, a physical parameter that is not related to the linguistic meaning of the voice signal, such as the pitch of the voice signal, the voice energy, etc. according to the input speech is extracted. Next, in the linguistic parameter extraction step 120, the speech signal is separated into units of phonemes by performing the speech recognition operation of the speech recognition unit. It will extract scientifically significant parameters. In the emotional state calculating step 130, various parameters extracted in the non-verbal parameter extraction step 110 and the linguistic parameter extraction step 120 are synthesized, and the weights are appropriately assigned to each of the plurality of parameters to maximize the speaker's emotional state. The final emotional state coefficient is calculated.

In this embodiment, the linguistic parameter extraction step 120 is performed after the non-verbal parameter extraction step 110 is performed as shown in the drawing. However, the above two steps may be reversed. You can also make these two steps run in parallel at the same time.

2 is a flow chart illustrating the flow chart of FIG. 1 in more detail.

In the present embodiment, the non-verbal parameter extraction step 21 includes four non-verbal, including pitch average value extraction step 210, pitch variance extraction step 220, voice energy average value extraction step 230, and voice energy dispersion extraction step 240. Including four extraction steps for extracting parameters, the linguistic parameter extraction step 22 extracts two linguistic parameters, such as a phoneme length average value extraction step 250 and a speech recognition result extraction step 260. It is intended to include two extraction steps.

First, the pitch average value extraction step 210 will be described.

Many methods have been proposed to calculate the average pitch of a speech signal. However, due to the inherent nature of speech, the pitch can be measured only in the voiced part and not in the unvoiced part. Therefore, in order to increase the reliability of the pitch of the extracted voice signal, a task of dividing the voice signal part from the voiced sound part must be performed. In the present invention, the pitch is estimated by using a normalized autocorelation function method, and a new method of classifying the speech signal into the voiced and unvoiced parts has been devised to greatly increase the reliability of the average pitch extraction of the speech signal. Improved.

In the normalized autocorrelation function method, once the speech signal is divided into short intervals of about 20 ms, the normalized autocorrelation function values for all possible pitches are calculated for each interval, and the pitch that maximizes the function value is calculated. Find and determine the pitch of the voice signal. At this time, each section divided about 20ms is called a frame, and when the total number of frames is N and the pitch of the nth frame is P (n), the average value of the pitch of the input voice signal Is determined as follows.

However, when the average pitch value is actually obtained, the pitches incorrectly obtained in the section corresponding to the unvoiced portion, such as some consonant sections, affect the overall accuracy, so the present invention is to obtain the average pitch value in the following manner.

In the above formula, v (n) is 1 if the nth frame is a voiced frame, and 0 if it is an unvoiced frame. For v (n), the pitch value of the previous 3 frames including the present, that is, P (n) If the amount of change in P (n-1) or P (n-2) is less than the experimentally determined threshold (for example, 10% of the current pitch), the interval is considered to be a relatively stable interval. It is treated as a voiced sound interval, that is, a section in which v (n) = 1. Similarly, if the change amount is larger than the threshold value, the pitch is regarded as an unstable section, and the section is treated as an unvoiced section, that is, a section in which v (n) = 0. This relationship is expressed as a formula below.

Next, the pitch dispersion extraction step 220 will be described.

Different people have different pitches. Therefore, it is difficult to set a predetermined criterion of pitch in order to recognize an emotional state for an unspecified number of speakers, and to use only such pitch as a criterion for determining the emotional state.

Therefore, the pitch is different from one person to another, but when calculating the average value of the pitch, using the amount of pitch change that determines how much the input voice signal changes from the average value as another parameter for determining the emotional state Very useful. It is because, by various studies and experiments, it is known that in general, when the emotion is furious, the pitch change of the speech is more rapid than usual.

In the emotion recognition method according to the present invention, once the average pitch value is calculated in the pitch average value extracting step 210, the change amount can be easily calculated by the variation of the pitch as follows.

Next, the voice energy average value extraction step 230 and the voice energy dispersion extraction step 240 will be described.

Unlike the pitch, energy is generally a parameter that is not related to voiced / unvoiced sound. However, in the present invention, since the parameter for the part where the change is distinct according to the emotional state is extracted, only the average voice energy in the voiced sound section is calculated to recognize the emotion. It is intended to be used as a parameter. In other words, the speaker in anger is generally louder than usual, and the energy of the unvoiced section does not change significantly, and only the energy of the voiced section occurs. Therefore, it is effective to calculate the average energy in the voiced sound interval.

Therefore, in the sound energy average value extraction step 230, the sound energy average value of the input voice signal is extracted by extracting the voice energy E (n) of the nth frame by the following equation.

In addition, as in the case of the pitch described above, in the case of the voice energy, it is effective to obtain not only the average value but also the change amount of the voice energy, that is, the variance, and use it as another parameter of the emotional state recognition. In 240, the dispersion of speech energy was extracted by the following equation.

Next, the phoneme length average value extraction step 250 and the speech recognition result extraction step 260 included in the linguistic parameter extraction step 22 will be described.

The phoneme length average value extracting step 250 extracts an average value of phoneme lengths from an input voice signal, and the average value of the phoneme lengths is measured to calculate a speaker's speech rate. This is because speakers who are in a state of emotion generally tend to speak faster than usual.

There may be several ways to estimate the speech rate of speech. The simplest thing is to memorize the utterance length (duration time) of the speech signal that has been uttered in advance for the same utterance in terms of contents, and use it as a reference for judging the utterance rate for subsequent utterances. However, this is very unlikely to be practical in that the content to be uttered must be determined in advance. In order to solve this problem, the present invention proposed a new method for measuring the speech length of speech. That is, the speech is recognized by the speech recognition module, and based on the words recognized by the speech recognition module, the phonetic sequence of the phonetic symbols of Korean corresponds to the actual speech, and then the average speech time for each phoneme sequence is obtained. will be. After that, the final average value of the average uttering time of all the phonemes appearing is used as a parameter representing the speech uttering speed.

In order to calculate the speech utterance in the example of the emotion recognition system using the voice according to the present invention, a total of 47 phoneme sequences as shown in Table 1 below may be used as the phonetic sequence of Korean phonetic symbols.

Initiality A ㄲ N C ㄸ D M ㅂ ㅃ S Initiality neutrality ㅆ ㅇ ㅈ ㅉ H Lol ㅌ ㅍ ㅎ ㅏ neutrality ㅐ ㅑ ㅒ ㅓ ㅔ ㅕ ㅖ ㅗ ㅘ ㅙ neutrality ㅚ ㅛ TT ㅝ ㅞ ㅟ ㅠ ㅡ ㅢ ㅣ Jongseong A N C D M ㅂ ㅇ

TABLE 1

On the other hand, the present inventors have generally found that the part that actually changes when the rate of ignition changes is the neutral part, and the lengths of the initial and the final part have characteristics that change little or without distinct rules. Therefore, in order to reflect this characteristic, when the average speech duration of the nth phoneme is T (n), the average value D of phoneme lengths is calculated as shown in the following equation.

That is, the average length of the phoneme is the weight of each phonetic phoneme constituting the speech signal (for example, the weight α (n) is 0.2 if the nth phoneme is initial or final and 1.0 if neutral). The sum of the product multiplied by the " multiplied by " is divided by the number (eg, 47) of the phonetic phoneme stored in the speech recognition module.

Therefore, the average value of the phoneme length can be extracted by further weighting the average of the average neutral utterance time, and the average utterance time thus obtained is obtained from the average value of the utterance time measured in the phoneme unit. It has the advantage of being able to accurately measure the relative speed of ignition regardless of whether it is already stored.

In this embodiment, the speech recognition module for measuring the average speech time may be used a multi-mixture system based on the continuous hidden Markov model (HMM), the speech recognition is 16 mix It can be configured based on a phone (16 mixture triphone). On the other hand, the process of mapping the phonetic sequence of phonetic symbols with the speech signal actually spoken may be performed by the Viterbi algorithm.

Next, the voice recognition result extraction step 260 will be described.

The prior art emotion recognition method is theoretically based on Fukuda's research that "humans can estimate the emotional state to some extent only by the non-verbal information of the speaker's speech itself even when no language information is given at all." Since most of the development was based on the grounds, the contents of the speech were not taken into account.

However, there are times when it is possible to estimate the emotional state of a person who is actually speaking even by the voice-recognized result, for example, a word such as "Hey!" Is in a more emotional state than in the speaker's normal emotional state. You can think of it as a word with a higher probability of being used.

Thus, a list of specific words suggesting that the emotions are frenzy is provided, speech recognition of the speaker's utterances is performed to extract words corresponding to the specific words list, and each word can be used in the frenzyed emotional state. If you set the probability, in addition to non-verbal parameters, it can be used as a new parameter that can improve the accuracy of emotion recognition.

Although the above list of words may vary from case to case, it is also possible to create some common lists. Table 2 below illustrates some of these word lists.

word Chance to be used in a raging emotional state (variable) Hey! 1.0 you! 1.0 change! 0.8 Quickly 0.7 Do you want to change it? 0.7 why 0.6 mess 0.8 Etc 0.0

TABLE 2

3 is a flowchart illustrating a process of calculating an emotional state of a speaker in an emotional state calculating step by combining measured values of each parameter in an example of an emotion recognition method using voice according to the present invention.

In the emotion recognition method using the voice according to the present invention, in order to effectively sum up all six pieces of information such as average pitch, pitch variance, average voice energy value, variance of voice energy, average length of phoneme, and probability value for a specific word, the state of normal and furious After collecting a large number of speech databases about emotional states and analyzing their statistical characteristics, the final emotional state extraction algorithm was defined.

The pitch average value of the input voice signal , The average of the negative energy Pitch dispersion dispersion Voice energy dispersion , The phoneme length average , The probability value for a particular word The average pitch value in the normal state obtained from a large amount of voice data , The average of the negative energy Pitch dispersion Voice energy dispersion , The phoneme length average , The final emotional state coefficient AER is calculated as follows.

In the above equation, a0 to a6 are weights for each parameter and can be changed according to the characteristics of the actual voice service.

Once the final emotional state coefficient AER is calculated, the emotional state can be extracted according to the value of the AER. When it is necessary to divide into two stages of normal state and anger state, one threshold value is set, and when the AER is smaller than the threshold value, it is determined as the normal state, and when it is larger than the threshold value, it is determined that it is anger state. Can be. Alternatively, in the case of dividing into three emotional states as shown in FIG. 3, two threshold values are set, and when the final emotional state coefficient AER is smaller than the first threshold value, the normal state is assumed. If it is larger and smaller than the second threshold, it may be determined to be somewhat excited, and if it is larger than the second threshold, it may be determined to be angry. Alternatively, it may be configured to further refine the emotional state in another embodiment.

4 is a block diagram showing an example of an emotion recognition system using voice according to the present invention.

The emotion recognition method using the voice according to the present invention may be performed on the emotion recognition system 40 using the voice as shown in FIG. 4.

That is, the speech input is performed by receiving the speaker's speech from the input unit 400, and the signal processor 420 extracts the pitch average value, pitch variance, speech energy average value, and speech energy variance from the input speech signal. The speech recognition unit 430 extracts a specific word according to the phoneme length average value and the speech recognition result. The controller 410 performs an overall operation of the input unit 400, the signal processor 420, the voice recognizer 430, the database 440, and the output unit 450, and calculates a final emotional state coefficient AER. Do it. The database 440 stores a large amount of voice data about average pitch, average voice energy, pitch variance, and variance of voice energy for the normal state and the anger state, and also recognizes the voice recognition of the voice recognition unit 430. Phonetic symbol used for storing the phoneme data. The output unit 450 outputs a signal of an appropriate type for transmitting information on the emotional state according to the final emotional state coefficient calculated by the controller 410 to another system that needs the information.

5 is a block diagram illustrating an example of a call center counselor connection system using an emotion recognition system using voice according to the present invention.

The emotion recognition system using the voice according to the present invention may be more preferably implemented for a purpose such as greatly improving the performance of the call center system in addition to an existing call center system or the like, rather than configuring an original system by itself.

In the embodiment shown in Figure 5, a private branch exchange (510) connected to the customer's telephone line 500, the CTI server 520 for controlling the overall operation of the call center, and the emotion recognition system embedded The emotion recognition server 540, an answering machine 530 connected to the CTI server, and at least one counselor telephone line 550 connected to the CTI server are included.

When the emotion recognition server 540 recognizes the emotion state of the customer by extracting and combining the various parameters as described above from the voice signal according to the speech of the customer, for example, when the customer is in anger state The CTI server 520 switches the call connection of the customer's telephone line from the automatic answering server 530 to the counselor's telephone line 550, which is a man-to-man type of counselor. Complaint work can be carried out. Alternatively, it is possible to improve the consultation efficiency and customer satisfaction with the customer by switching the call connection from one telephone line for a counselor connected to another to a telephone line in which a more skilled counselor is in charge.

In the present embodiment, the case of applying the emotion recognition system using the voice according to the present invention to the call center system using the general telephone network, but this is just an example, a call center system using the Internet network, various IT solutions requiring other customer service In the agent connection service, the service automatically performs the emotion recognition from the voice of the customer, and from this it can be applied to all services that automatically connect the appropriate agent for the situation.

When using the emotion recognition system and emotion recognition method using the voice according to the present invention as described above, in order to effectively estimate and recognize the speaker's emotional state, verbal information and non-verbal information indicating the emotional state of the speaker included in the speaker's speech An emotion recognition system and an emotion recognition method using voice for extracting and comprehensively analyzing a plurality of parameters related to information may be implemented.

Claims (14)

As a emotion recognition method using voice, A speech input step of receiving a voice signal according to the speaker's speech; Non-verbal parameter extraction step of extracting at least one non-verbal parameter from the spoken sound; Linguistic parameter extraction step of extracting at least one linguistic parameter according to the meaning of the spoken sound by speech recognition of the speech signal; And an emotional state calculating step of finally calculating the emotional state of the speaker by combining the linguistic parameters and the non-verbal parameters. The method of claim 1, The non-verbal parameter includes an average value of pitches of the speech signal obtained by dividing the speech signal into speech signal frames at predetermined time intervals, and dividing the sum of the pitches of the voiced frames among the frames by the number of the voiced frames. Emotion recognition method using a voice, characterized in that. The method of claim 1, The nonverbal parameter includes a variance of the pitch of the speech signal obtained by dividing the sum of the squares of the difference between the mean value of the pitch and the pitch in each of the voiced frames by the number of frames of the voiced sounds. Emotion recognition method using a voice, characterized in that. The method of claim 1, The non-verbal parameter is an average value of the speech energy of the speech signal obtained by dividing the speech signal into speech signal frames at predetermined time intervals, and dividing the total of speech energy in the frames, which are voiced sounds, by the number of frames, which are voiced sounds. Emotion recognition method using a voice comprising a. The method of claim 1, The non-verbal parameter is a variance of the speech energy of the speech signal obtained by dividing the sum of the squares of the difference between the mean value of the speech energy and the speech energy in each of the voiced frames by the number of frames of the voiced sounds. Emotion recognition method using a voice comprising a. The method of claim 1, The linguistic parameter includes a speech rate of the speech signal. The method of claim 6, And said speech rate of said speech signal is a relative velocity of an average value of actual phoneme lengths of said speech signal to an average value of standard phoneme lengths of said speech signal. The method of claim 7, wherein The average value of the standard phoneme length of the voice signal may include the phoneme of each Korean phonetic phoneme constituting the voice signal recognized by the voice recognition module. Emotion recognition method using speech, characterized in that the average value of the length. The method of claim 7, wherein The average value of the actual phoneme length of the speech signal is obtained by dividing the sum of the lengths of the phonetic phonemes constituting the voice signal by the weight of the phonetic phonemes by the number of the phonetic phonemes stored in the speech recognition module. Emotion recognition method using speech, characterized in that the value. The method of claim 9, The weight of the phonetic phoneme of the neutral among the phonetic phonemes is higher than the weights of the initial and the final voice of the phonetic phoneme. The method of claim 1, The linguistic parameter may include an emotion calculation coefficient corresponding to each special word included in a recognition result of the speech signal by the speech recognition module. The method of claim 1, The emotional state calculating step, The difference between the actual pitch mean value and the standard pitch mean value divided by the standard pitch mean value, multiplied by a first weight, A difference obtained by dividing the difference between the average value of the actual voice energy and the average value of the standard voice energy by the average value of the standard voice energy, multiplied by a second weight, The difference between the standard deviation of the actual pitch and the standard deviation of the standard pitch divided by the standard deviation of the standard pitch mean, multiplied by a third weight, The difference between the standard deviation of the actual voice energy and the standard deviation of the standard voice energy divided by the standard deviation of the standard voice energy, multiplied by a fourth weight, The difference between the average value of the actual phoneme length and the average value of the reference phoneme length divided by the average value of the reference phoneme length is multiplied by a fifth weight value, The sixth weight multiplied by the emotion calculation coefficient corresponding to the special word included in the result of speech recognition And calculating the emotional state of the speaker by comparing the final emotional state coefficient calculated by summing with a predetermined threshold. An emotion recognition system using a voice including a voice signal input unit, a signal processor, a voice recognition unit, a controller and an output unit, The voice signal input unit converts a speaker's voice into an electrical signal and transmits the voice signal to at least one of the signal processor and the voice recognizer. The signal processing unit may include: a pitch average value extracting unit extracting an average value of pitches of the voice signal, a pitch variation extracting unit extracting a degree of change of the average value of the pitches, and an average value of voice energy of the voice signal; Further comprising a voice energy average value extraction unit, and a voice energy dispersion extraction unit for extracting the degree of change of the average value of the voice energy, The speech recognition unit further includes a speech rate extracting unit for extracting a speech rate of the speech signal and a speech recognition result extracting unit for extracting a specific word included in the speech signal. The control unit uses the at least one parameter of the pitch average value, the pitch variance, the speech energy average value, the speech energy dispersion, the speech rate, and the speech recognition result calculated by the signal processing unit and the speech recognition unit to determine the final feeling. Calculating a recognition coefficient and comparing the final emotion recognition coefficient with a predetermined threshold to recognize the emotional state of the speaker. A call center agent connection system using the emotion recognition system according to claim 13, A private branch exchange connected to the customer's telephone line, a CTI server that controls the overall operation of the call center, an emotion recognition server with a built-in emotion recognition system, an answering machine connected to the CTI server, and at least a connection to the CTI server. Includes one agent phone line, When the emotion recognition server recognizes the emotion state of the customer from the voice signal according to the customer's utterance, the CTI server according to the result of the call connection of the customer's telephone line from the answering machine to the telephone line for the counselor Call center agent connection system using an emotion recognition system to switch to or transfer a call connection from one agent telephone line to another.