CN110956981B - Speech emotion recognition method, device, device and storage medium - Google Patents

Abstract

The invention belongs to the technical field of voice signal processing and mode recognition, and discloses a voice emotion recognition method, a device, equipment and a storage medium. The method comprises the following steps: acquiring a test voice sample with a preset dimension, and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples; extracting signal characteristic data of the initial voice sample to obtain voice signal characteristic data to be processed; performing feature statistics on the feature data of the voice signal to be processed through a preset statistical function to obtain a feature statistical result to be confirmed; obtaining feature target data through a preset multi-target optimization algorithm according to the feature statistical result to be confirmed; and inputting the characteristic target data into a preset Softmax classification model to obtain a speech emotion recognition result. Through the method, the speech emotion fragments form speech emotion data, and the speech emotion data are input into the preset Softmax classification model, so that speech emotion can be better recognized.

Description

Speech emotion recognition method, device, device and storage medium

technical field

The present invention relates to the technical field of speech signal processing and pattern recognition, and in particular, to a speech emotion recognition method, device, device and storage medium.

Background technique

At present, there are many speech emotion recognition methods, but these methods do not pay attention to the short-term and locality of human speech emotion expression. For example, in speech emotion recognition, the first half of a sentence or a word of anger can be regarded as anger in the whole sentence. The following problems will arise: First, the use of whole sentences to identify emotions often dilutes the characteristic changes of emotions. For example, "We will go to Beijing tomorrow, do you think it's feasible?" The second half of this sentence often reflects a greater emotional difference. As a result, the use of temporal mean pooling, convolution, and fully-connected layers for all features in deep learning will dilute the feature changes of emotions; second, when locally combined into sentences, the feature changes of emotions are often neutralized. As we all know, Chinese intonation has one to four tones, and the second and fourth tones have completely opposite characteristics in time change. This leads to the use of time-based mean pooling in deep learning, and the attention layer for time series will neutralize the feature changes of emotions; 3. The position of the words that make up the emotion is not fixed in the sentence, which will cause the characteristics of sympathy big difference. For example, "Is this possible?" and "Is this possible? This!" express the same meaning, but the output features of existing convolutional neural networks are completely different.

The above content is only used to assist the understanding of the technical solutions of the present invention, and does not mean that the above content is the prior art.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide a speech emotion recognition method, device, equipment and storage medium, aiming at solving the technical problem of how to accurately speech emotion.

To achieve the above object, the invention provides a kind of speech emotion recognition method, and described method may further comprise the steps:

Acquire a test voice sample of a preset dimension, and perform segmentation processing on the test voice sample according to a preset rule to obtain a plurality of initial voice samples;

Extracting the signal feature data of the initial voice sample to obtain the feature data of the voice signal to be processed;

Perform feature statistics on the feature data of the speech signal to be processed by using a preset statistical function to obtain a feature statistics result to be confirmed;

According to the statistical result of the feature to be confirmed, the feature target data is obtained through a preset multi-objective optimization algorithm;

Inputting the feature target data into a preset Softmax classification model to obtain a speech emotion recognition result.

Preferably, before the step of obtaining a test voice sample of a preset dimension, and performing segmentation processing on the test voice sample according to a preset rule, and obtaining a plurality of initial voice samples, the method further includes:

Acquiring training voice samples of preset dimensions, and performing segmentation processing on the test voice samples through preset rules to obtain a plurality of initial training voice samples;

Perform feature extraction on the initial training voice sample to obtain the features of the training voice signal to be processed;

Perform feature statistics on the features of the to-be-processed training speech signal by using a preset statistical function, and obtain a statistical result of the to-be-confirmed training features;

According to the statistical result of the training feature to be confirmed, the target training feature data is obtained through a preset multi-objective optimization algorithm;

Obtain the emotion category corresponding to the target training feature data according to the target training feature data;

A preset Softmax classification model is established according to the emotion category and target training feature data corresponding to the emotion category.

Preferably, the step of obtaining target training feature data through a preset multi-objective optimization algorithm according to the statistical results of the training features to be confirmed includes:

Perform emotional classification on the statistical results of the training features to be confirmed, and obtain training feature data to be optimized corresponding to different emotional categories;

According to the training feature data to be optimized, target training feature data is obtained through a preset multi-objective optimization algorithm.

Preferably, the step of inputting the feature target data into a preset Softmax classification model to obtain a speech emotion recognition result includes:

Inputting the feature target data into the preset Softmax classification model to obtain voice emotion category data;

performing data statistics on the voice emotion category data to obtain a voice emotion category data value;

A speech emotion recognition result is obtained according to the speech emotion category data value.

Preferably, the step of obtaining a speech emotion recognition result according to the speech emotion category data value includes:

Judging whether the voice emotion category data value belongs to the preset voice emotion category threshold range;

If the voice emotion category data value belongs to the preset voice emotion category threshold range, a voice emotion recognition result is obtained according to the voice emotion category data value.

Preferably, after the step of judging whether the voice emotion category data value belongs to the preset voice emotion category threshold range, the step further includes:

If the voice emotion category data value does not belong to the preset voice emotion category threshold range, returning to the step of inputting the feature target data into the preset Softmax classification model to obtain voice emotion category data.

Preferably, the step of performing feature statistics on the feature data of the to-be-processed voice signal by using a preset statistical function to obtain the feature statistics result to be confirmed includes:

Screening the feature data of the to-be-processed voice signal to obtain label sample feature data;

Feature statistics are performed on the label sample feature data through a preset statistical function to obtain a feature statistics result to be confirmed.

In addition, in order to achieve the above object, the present invention also provides a voice emotion recognition device, the device includes: an acquisition module for acquiring test voice samples of preset dimensions, and classifying the test voice samples according to preset rules Segment processing to obtain multiple initial speech samples;

an extraction module, used for extracting the signal feature data of the initial voice sample to obtain the feature data of the voice signal to be processed;

a statistical module, configured to perform feature statistics on the feature data of the to-be-processed voice signal through a preset statistical function, and obtain a feature-to-be-confirmed statistical result;

a calculation module, configured to obtain characteristic target data through a preset multi-objective optimization algorithm according to the statistical result of the to-be-confirmed characteristic;

A determination module, configured to input the feature target data into a preset Softmax classification model to obtain a speech emotion recognition result.

In addition, in order to achieve the above object, the present invention also proposes an electronic device, the device includes: a memory, a processor, and a speech emotion recognition program stored in the memory and running on the processor, the speech The emotion recognition program is configured to implement the steps of the speech emotion recognition method as described in any of the above.

In addition, in order to achieve the above object, the present invention also provides a storage medium, on which a speech emotion recognition program is stored, and when the speech emotion recognition program is executed by a processor, the speech emotion as described in any one of the above is realized. Identify the steps of the method.

The present invention obtains the test voice samples with preset dimensions firstly, and performs segmentation processing on the test voice samples according to the preset rules to obtain a plurality of initial voice samples, and then extracts the signal feature data of the initial voice samples to obtain the to-be-processed voice signal. feature data, and filter the feature data of the to-be-processed voice signal to obtain the feature data of the label sample, perform feature statistics on the feature data of the label sample through a preset statistical function, obtain the feature statistics result to be confirmed, and then according to the feature to be confirmed Statistical results are obtained through a preset multi-objective optimization algorithm to obtain characteristic target data, and finally the characteristic target data is input into a preset Softmax classification model to obtain a speech emotion recognition result. Through the above method, speech emotion segments and the emotional relationship between sentences and segments can be fully utilized, and converted into speech emotion data, thereby improving the speech emotion recognition effect.

Description of drawings

1 is a schematic structural diagram of an electronic device of a hardware operating environment involved in an embodiment of the present invention;

2 is a schematic flowchart of a first embodiment of a speech emotion recognition method according to the present invention;

3 is a schematic flowchart of a second embodiment of a speech emotion recognition method according to the present invention;

FIG. 4 is a structural block diagram of a first embodiment of a speech emotion recognition apparatus according to the present invention.

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Referring to FIG. 1 , FIG. 1 is a schematic structural diagram of an electronic device of a hardware operating environment involved in an embodiment of the present invention.

As shown in FIG. 1 , the electronic device may include: a processor 1001 , such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 . Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or may be a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Those skilled in the art can understand that the structure shown in FIG. 1 does not constitute a limitation on the electronic device, and may include more or less components than the one shown, or combine some components, or arrange different components.

As shown in FIG. 1 , the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module and a speech emotion recognition program.

In the electronic device shown in FIG. 1, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in In the electronic device, the electronic device invokes the speech emotion recognition program stored in the memory 1005 through the processor 1001, and executes the speech emotion recognition method provided by the embodiment of the present invention.

An embodiment of the present invention provides a speech emotion recognition method. Referring to FIG. 2 , FIG. 2 is a schematic flowchart of a first embodiment of a speech emotion recognition method of the present invention.

In this embodiment, the speech emotion recognition method includes the following steps:

Step S10: Acquire a test voice sample of a preset dimension, and perform segmentation processing on the test voice sample according to a preset rule to obtain a plurality of initial voice samples.

It should be noted that, before the steps of obtaining a test voice sample of a preset dimension, and performing segmentation processing on the test voice sample according to a preset rule, and obtaining a plurality of initial voice samples, it is necessary to obtain a training voice sample of a preset dimension. , and perform segmentation processing on the test voice samples through preset rules to obtain a plurality of initial training voice samples, perform feature extraction on the initial training voice samples, and obtain the characteristics of the training voice signal to be processed. Perform feature statistics on the characteristics of the training voice signal to be processed, obtain the statistical results of the training characteristics to be confirmed, obtain target training characteristic data through a preset multi-objective optimization algorithm according to the statistical results of the training characteristics to be confirmed, and obtain the target training characteristic data according to the target training characteristic data Obtain the emotion category corresponding to the target training feature data, and establish a preset Softmax classification model according to the emotion category and the target training feature data corresponding to the emotion category.

In addition, it should be understood that the above-mentioned preset rules are user-defined sample division rules, that is to say, if the acquired test voice samples of the preset dimension have a corresponding duration of 5s, the preset rules are set as 0.2s, then 25 segments of initial speech samples of 0.2s are obtained after division according to preset rules.

In addition, it should be noted that the preset dimension mentioned above may be a time dimension or a non-time dimension, etc., which is not limited in this embodiment.

Step S20 : extracting the signal feature data of the initial speech sample to obtain the feature data of the speech signal to be processed.

In addition, it should be understood that, performing signal feature data extraction on the initial speech sample to extract Mel Frequency Cepstrum Coefficients (MFCC), Log Frequency Power Coefficients (LFPC), and Linear Prediction Cepstrum Coefficients (Linear Predictive Cepstral Coding, LPCC), Zero Crossing with Peak Amplitude (ZCPA), Perceptual Linear Predictive (Perceptual Linear Predictive, PLP), Rasta Filter Perceptual Linear Predictiv (Rasta Perceptual Linear Predictiv, R-PLP) ).

It should be understood that the feature extraction result of each type of feature mentioned above is a two-dimensional matrix, one of which is the time dimension, and then the first-order derivative ΔF _i and the second-order derivative of each type of feature F _i on the time dimension are calculated. ΔΔF _i , and concatenate the original features, first-order derivative results, and second-order derivative results in the non-time dimension to form the final feature extraction result of each type of feature; Concatenation in dimension is the feature extraction result of the sample.

In addition, for ease of understanding, the following examples are provided:

Suppose, the corresponding F _MFCC ∈ R ^39×z , ΔF _MFCC ∈ R ^39×z , ΔΔF _i ∈ R ^39×z , where z is the number of frames, that is, the number of time dimensions, the concatenation result in the non-time dimension

串接后为

When MFCC and LPCC are connected, if

concatenated as

In addition, it should be understood that MFCC, LFPC, LPCC, ZCPA, PLP, R-PLP features are extracted during each speech signal feature extraction, wherein the number of Mel filters of MFCC and LFPC is 40; LPCC, PLP, The linear prediction orders of R-PLP are 12, 16, and 16 respectively; the frequency segments of ZCPA are: 0, 106, 223, 352, 495, 655, 829, 1022, 1236, 1473, 1734, 2024, 2344, 2689 , 3089, 3522, 4000. Therefore, the dimensions of each type of feature of each statement are: ti*39, ti*40, ti*12, ti*16, ti*16, ti*16, where ti is the frame number of the i-th statement, and the multiplication sign The following numbers are the dimensions of each frame feature. In order to obtain the change of the speech signal in the time dimension, the first-order derivative and the second-order derivative are also calculated for the above-mentioned features in the time dimension. Finally, the dimensions of each type of feature are: ti*117, ti*140, ti*36, ti*48, ti*48, ti*48. The extracted speech signal feature of the ith sample is composed of all the above features, and the dimension is ti*(117+140+36+48+48+48).

Step S30: Perform feature statistics on the feature data of the to-be-processed voice signal by using a preset statistical function to obtain a feature statistics result to be confirmed.

It should be noted that the statistical functions are used to obtain the above features in the time dimension by using the mean, standard deviation, minimum value (min), maximum value (max), kurtosis (kurtosis) and skewness (skewness). Statistical results above.

In addition, it should be understood that the characteristic data of the label samples are obtained by screening from the statistical results obtained above, and the characteristic statistics of the label samples are carried out through a preset statistical function to obtain the statistical results of the characteristics to be confirmed, and there will be The feature statistics of labeled samples are recorded as {x ₁ , x ₂ ,...,x _n }, where n is the number of labeled samples.

Step S40: According to the statistical result of the feature to be confirmed, obtain feature target data through a preset multi-target optimization algorithm.

In addition, it should be noted that {x ₁ ,x ₂ ,...,x _n } in the previous step is divided into X _A =[x ₁ ,x ₂ ,...,x _m ] according to the label of the statement, X _B =[x _m+1 ,x _m+2 ,...,x _n ], where X _A is a segment of type A emotion, and X _B is a segment of type B emotion, training a sentence segment emotion classification method based on propensity cognitive learning Proceed as follows:

(1) For x∈X _A , divide the angle between the sample in the Parzen window centered on x in X _A and the center sample x into multiple bins, and then use the following formula to calculate the distribution characteristics of the data around x in X _A .

b _x =[b ₁ ,b ₂ ,...,b _k ]

where b _j represents the jth box, 1(x _i ∈ X _j ) is 1 when x _i belongs to X _j , otherwise it is 0, X _j is a subset of X _A , and the difference between the samples in X _j and x is The angles between are distributed in the jth bin.

(2) For x∈X _A , divide the angle between the sample in the Parzen window centered on x in X _B and the center sample x into multiple bins, and then use the following formula to calculate the distribution characteristics of the data around x in X _A .

表示第

个箱子，1(x_i∈X_j)在x_i属于X_j时的值为1否则为0，X_j是X_B的子集，X_j内的样本与x之间的角度分布在第j个箱子中。in the formula

means the first

box, 1(x _i ∈ X _j ) is 1 when _xi belongs to X _j , otherwise 0, X _j is a subset of X _B , and the angle between the samples in X _j and x is distributed in the jth in a box.

(3) Use the following formula to calculate the difference in the data distribution of the two data sets near the x point:

表示两向量之间的距离，此处使用欧氏距离。in the formula

Represents the distance between two vectors, where Euclidean distance is used.

倾向于B情感的片段集合

以及倾向于中性情感的片段集合

其中

为d_x>T的x组成的集合。

为d_x<-T的x组成的集合。

为T>d_x>-T组成的集合。T是自主设置的阈值。对每个集合，再使用谱聚类的方法聚成多个区域，得到每个片段xi的区域标签

(4) According to the calculation result of the previous step, the set of fragments tending to A emotion can be obtained

A collection of snippets leaning towards B sentiment

and a collection of clips that tend towards neutral emotions

in

is the set of x where d _x > T.

is the set of x where d _x <-T.

is a set consisting of T>d _x >-T. T is an autonomously set threshold. For each set, the spectral clustering method is used to cluster into multiple regions, and the region label of each segment xi is obtained.

L＝[L_A,L_B,L_C]，其中L_A∈R^p、L_B∈R^q、L_C∈R^u，p、q和u分别为

和

样本的个数，L_A、L_B和L_C中的元素值分别为1、2、3。使用下述目标方程，学习片段的特征子空间：(5) Definition

L=[L _A , L _B , L _C ], where L _A ∈ R ^p , L _B ∈ R ^q , L _C ∈ R ^u , p, q and u are respectively

and

The number of samples, the element values in L _A , L _B and L _C are 1, 2, and 3, respectively. The feature subspace of the segment is learned using the following objective equation:

J=J ₁ (o _i ,o _j )+β*J ₂ (o _i ,o _j )

和

三个类之间的类内距离较小，类间距离较大，定义如下：β is the equilibrium parameter. where J ₁ (o _i ,o _j ) can realize

and

The intra-class distance between the three classes is small, and the inter-class distance is large, which is defined as follows:

和

映射到子空间之后的结果。l_i和l_j对应o_i和o_j在L中的值。m是一个阈值，调整类间距离的范围，本发明中取1。G_ij为xi和x_j之间的高斯距离。计算公式如下：where o _i and o _j are

and

The result after mapping to the subspace. l _i and l _j correspond to the values of o _i and o _j in L. m is a threshold, and the range of the distance between classes is adjusted, which is taken as 1 in the present invention. G _ij is the Gaussian distance between xi and x _j . Calculated as follows:

J2(o _i ,o _j ) can try to keep the relative relationship in each region unchanged, and the regions belonging to the same class are relatively close, but do not overlap. Defined as follows:

和

是xi和xj的区域标签。G_li是l_i类所有G_ij中的最大值。实现当两个片段属于同一区域时，保持他们之间的关系，当两者不属于同一区域但是属于同一类别时，以一个小的权重最小化他们之间的距离，可使两个区域尽量不重叠。in the formula

and

are the region labels for xi and xj. G _li is the maximum value _{among all G ij of} class li. Realize that when two fragments belong to the same area, keep the relationship between them, and when the two do not belong to the same area but belong to the same category, minimize the distance between them with a small weight, so that the two areas can be as different as possible. overlapping.

和

可得到W₁,W₂,…,W_q和b₁,b₂,…,b_q的值，

是求J对W的导数，

是求J对b的导数。To optimize the objective equation J, we define o _i =φ(W _q φ(…W ₃ φ(W ₂ φ(W ₁ x _i +b ₁ )+b ₂ )+b ₃ )+b ₄ ) where φ( ·) is the sigmoid function, W ₁ , W ₂ ,...,W _q are the mapping matrices, and b ₁ ,b ₂ ,...,b _q are the offsets. by asking

and

The values of W ₁ , W ₂ ,...,W _q and b ₁ ,b ₂ ,...,b _q can be obtained,

is the derivative of J with respect to W,

is the derivative of J with respect to b.

Step S50: Input the feature target data into a preset Softmax classification model to obtain a speech emotion recognition result.

In addition, it should be understood that, according to the above steps to obtain W ₁ , W ₂ ,...,W _q and b ₁ ,b ₂ ,...,b _q , calculate the features of {x ₁ ,x ₂ ,...,x _m } Select result z.

In addition, it should be noted that the above-mentioned W ₁ , W ₂ ,...,W _q and b ₁ ,b ₂ ,...,b _q are characteristic target data in this application.

Furthermore, it should be understood that the speech emotion categories { _{l 1 , l 2 ,} ..., x _m } are respectively obtained using the preset Softmax classifier obtained during the training process. l _m }. Then vote for the sentiment of the statement according to {l ₁ ,l ₂ ,...,l _m }.

In addition, it should be noted that the feature target data is input into the preset Softmax classification model to obtain voice emotion category data, and data statistics are performed on the voice emotion category data to obtain the voice emotion category data value. The speech emotion category data value is used to obtain speech emotion recognition results.

In addition, the above-mentioned step of obtaining the voice emotion recognition result according to the voice emotion category data value is to judge whether the voice emotion category data value belongs to the preset voice emotion category threshold range, and if the voice emotion category data value belongs to all If the preset voice emotion category threshold range is selected, the voice emotion recognition result is obtained according to the voice emotion category data value; if the voice emotion category data value does not belong to the preset voice emotion category threshold value range, return the The feature target data is input into the preset Softmax classification model to obtain the voice emotion category data.

In addition, it should also be noted that the corpus used in the effect evaluation of emotion recognition of the present invention is a standard database in the field of speech emotion recognition. The training process is completed first, followed by the recognition test. The test pattern is performed in a 5-fold crossover. It can identify 7 emotions of anger, fear, irritability, disgust, happiness, neutrality, and sadness. The average classification accuracy rate is 94.65% in the case of speaker dependence. Except for happiness and anger, which are more easily confused, the degree of discrimination between other emotions better. The average classification accuracy is 89.30% in the case of speaker independence.

In this embodiment, a test voice sample of a preset dimension is obtained first, and a plurality of initial voice samples are obtained by segmenting the test voice sample according to a preset rule, and then signal feature data is extracted for the initial voice sample to obtain the voice to be processed. signal feature data, and filter the feature data of the to-be-processed voice signal to obtain label sample feature data, perform feature statistics on the label sample feature data through a preset statistical function, obtain feature statistics results to be confirmed, and then The statistical results of the training features to be confirmed are divided into emotion categories, and the training feature data to be optimized corresponding to different emotion categories is obtained. According to the training feature data to be optimized, the target training feature data is obtained through a preset multi-objective optimization algorithm, and finally the feature target data Input into the preset Softmax classification model to obtain speech emotion recognition results. Through the above method, it is possible to make full use of speech emotional fragments and the emotional relationship between sentences and fragments to form a kind of tendency data, so that the process of human processing tendency can be simulated, and the unbalanced information of the data can be used to compare with each other. Constraints to separate segments of different sentiments, thereby increasing sample size and increasing sample diversity.

Referring to FIG. 3 , FIG. 3 is a schematic flowchart of a second embodiment of a speech emotion recognition method according to the present invention.

Based on the above-mentioned first embodiment, before the step S10, the speech emotion recognition method of this embodiment further includes:

Step S000: Acquire a training voice sample of a preset dimension, and perform segment processing on the test voice sample according to a preset rule to obtain a plurality of initial training voice samples.

Step S001: Perform feature extraction on the initial training voice sample to obtain the features of the training voice signal to be processed.

Step S002: Perform feature statistics on the features of the to-be-processed training speech signal by using a preset statistical function, and obtain a statistical result of the to-be-confirmed training features.

Step S003: Obtain target training feature data through a preset multi-objective optimization algorithm according to the statistical result of the training feature to be confirmed.

Step S004: Acquire an emotion category corresponding to the target training feature data according to the target training feature data.

Step S005: Establish a preset Softmax classification model according to the emotion category and target training feature data corresponding to the emotion category.

In addition, it should be noted that the above-mentioned step of obtaining target training feature data through a preset multi-objective optimization algorithm according to the statistical results of the training features to be confirmed is to classify the statistical results of the training features to be confirmed by emotion category , obtain training feature data to be optimized corresponding to different emotion categories, and obtain target training feature data through a preset multi-objective optimization algorithm according to the training feature data to be optimized.

In addition, it should be noted that the above-mentioned steps are to establish a preset Softmax classification model. In this stage, all speakers are trained separately to obtain the classifier corresponding to each speaker. The specific process is as follows:

Step (1-1) segment each statement;

Step (1-2) extracts the feature of each segment;

Step (1-3) performs feature statistics on all features;

Step (1-4) training a sentence fragment sentiment classification method based on tendency cognitive learning;

Step (1-5) trains a support vector machine for each feature subspace;

Step (1-6) The classification result is obtained by voting of the results of all support vector machines;

In addition, it should be noted that, in the step (1-1), the voice signal is segmented at intervals of 0.2 seconds.

In the step (1-2), the extracted speech signal features for each segment include: MFCC (Mel FrequencyCepstrum Coefficient, Mel frequency cepstrum coefficient), LFPC (Log Frequency Power Coefficients, logarithmic frequency power coefficient), LPCC ( Linear Predictive Cepstral Coding, linear prediction cepstral coefficient), ZCPA (Zero Crossing with Peak Amplitude, zero-crossing peak amplitude), PLP (Perceptual LinearPredictive, perceptual linear prediction), R-PLP (Ras ta Perceptual Linear Predictiv, Rasta Filter-aware linear prediction), the feature extraction result of each type of feature is a two-dimensional matrix, one of which is the time dimension; then the first-order derivative ΔF _i and the second-order derivative ΔΔF _i of each type of feature F _i in the time dimension are calculated. , and concatenate the original features, first-order derivative results, and second-order derivative results in the non-time dimension to form the final feature extraction result of each type of feature; the final feature extraction result of all the above-mentioned classes of features is in the non-time dimension. The concatenation is the feature extraction result of the sample.

The feature statistics of the features in the step (1-3) are: obtaining the statistical results of the mean, standard deviation, minimum value, maximum value, kurtosis, and skewness of the features in the time dimension, and the feature statistics results of the labeled samples are recorded. is {x ₁ , x ₂ ,...,x _n }, and the corresponding label is denoted as Y=[y ₁ , y ₂ ,..., y _n ]∈R ⁿ .

In the step (1-4), given a data set X _A =[x ₁ ,x ₂ ,...,x _m ], X _B =[x _m+1 ,x _m+2 ,...,x _n ], Among them, X _A is a segment of A-type emotion, and X _B is a segment of B-type emotion. The steps of training a sentence segment emotion classification method based on propensity cognitive learning are as follows:

Step (1-4-1) For x∈X _A , divide the angle between the sample in the Parzen window centered at x in X _A and the center sample x into multiple bins, and then use the following formula to calculate x in X _A Distribution characteristics of surrounding data.

b _x =[b ₁ ,b ₂ ,...,b _k ]

where b _j represents the jth box, 1(x _i ∈ X _j ) is 1 when x _i belongs to X _j , otherwise it is 0, X _j is a subset of X _A , and the difference between the samples in X _j and x is The angles between are distributed in the jth bin.

Step (1-4-2) For x ∈ X _A , divide the angle between the sample in the Parzen window centered at x in X _B and the center sample x into multiple bins, and then use the following formula to calculate x in X _A Distribution characteristics of surrounding data.

表示第

个箱子，1(x_i∈X_j)在x_i属于X_j时的值为1否则为0，X_j是X_B的子集，X_j内的样本与x之间的角度分布在第j个箱子中。in the formula

means the first

box, 1(x _i ∈ X _j ) is 1 when _xi belongs to X _j , otherwise 0, X _j is a subset of X _B , and the angle between the samples in X _j and x is distributed in the jth in a box.

Step (1-4-3) uses the following formula to calculate the difference in the data distribution of the two datasets near the x point:

表示两向量之间的距离，可使用多种距离计算方法。in the formula

Represents the distance between two vectors, and various distance calculation methods can be used.

倾向于B情感的片段集合

以及倾向于中性情感的片段集合

其中

为d_x>T的x组成的集合。

为d_x<-T的x组成的集合。

为T>d_x>-T组成的集合。T是自主设置的阈值。对每个集合，再使用谱聚类的方法聚成多个区域，得到每个片段xi的区域标签

Step (1-4-4) According to the calculation result of step (1-4-3), a set of fragments tending to A sentiment can be obtained

A collection of snippets leaning towards B sentiment

and a collection of clips that tend towards neutral emotions

in

is the set of x where d _x > T.

is the set of x where d _x <-T.

is a set consisting of T>d _x >-T. T is an autonomously set threshold. For each set, the spectral clustering method is used to cluster into multiple regions, and the region label of each segment xi is obtained.

L＝[L_A,L_B,L_C]，其中L_A∈R^p、L_B∈R^q、L_C∈R^u，p、q和u分别为

和

样本的个数，L_A、L_B和L_C中的元素值分别为1、2、3。使用下述目标方程，学习片段的特征子空间：Step (1-4-5) Definition

L=[L _A , L _B , L _C ], where L _A ∈ R ^p , L _B ∈ R ^q , L _C ∈ R ^u , p, q and u are respectively

and

The number of samples, the element values in L _A , L _B and L _C are 1, 2, and 3, respectively. The feature subspace of the segment is learned using the following objective equation:

J=J ₁ (o _i ,o _j )+β*J ₂ (o _i ,o _j )

和

三个类之间的类内距离较小，类间距离较大，定义如下：β is the equilibrium parameter. where J ₁ (o _i ,o _j ) can realize

and

The intra-class distance between the three classes is small, and the inter-class distance is large, which is defined as follows:

和

映射到子空间之后的结果。l_i和l_j对应o_i和o_j在L中的值。m是一个阈值，调整类间距离的范围。G_ij为x_i和x_j之间的高斯距离。计算公式如下：where o _i and o _j are

and

The result after mapping to the subspace. l _i and l _j correspond to the values of o _i and o _j in L. m is a threshold that adjusts the range of inter-class distances. G _ij is the Gaussian distance between x _i and x _j . Calculated as follows:

J ₂ (o _i ,o _j ) can try to keep the relative relationship in each region unchanged, and the regions belonging to the same class are relatively close, but do not overlap. Defined as follows:

和

是xi和xj的区域标签。G_li是l_i类所有G_ij中的最大值。实现当两个片段属于同一区域时，保持他们之间的关系，当两者不属于同一区域但是属于同一类别时，以一个小的权重最小化他们之间的距离，可使两个区域尽量不重叠。in the formula

and

are the region labels for xi and xj. G _li is the maximum value _{among all G ij of} class li. Realize that when two fragments belong to the same area, keep the relationship between them, and when the two do not belong to the same area but belong to the same category, minimize the distance between them with a small weight, so that the two areas can be as different as possible. overlapping.

和

可得到W₁,W₂,…,W_q和b₁,b₂,…,b_q的值，

是求J对W的导数，

是求J对b的导数。In order to optimize the objective equation J, define o _i =φ(W _q φ(…W ₃ φ(W ₂ φ(W ₁ x _i +b ₁ )+b ₂ )+b ₃ )+b ₄ ), where φ( ·) is the sigmoid function, W ₁ , W ₂ ,...,W _q are the mapping matrices, and b ₁ ,b ₂ ,...,b _q are the offsets. by asking

and

The values of W ₁ , W ₂ ,...,W _q and b ₁ ,b ₂ ,...,b _q can be obtained,

is the derivative of J with respect to W,

is the derivative of J with respect to b.

和

的特征子空间，训练Softmax分类器将情感A，情感B和中性情感C分开。Step (1-4-6) to step (1-4-5) obtained

and

The feature subspace of , trains a Softmax classifier to separate sentiment A, sentiment B and neutral sentiment C.

Step (1-4-7) According to the operation process of step (1-4-5) and step (1-4-6), train a softmax classifier capable of identifying all emotion pairs.

In addition, it should be understood that the following is a summary of the above:

Step 1: Segment all training sample speech at intervals of 0.2 seconds.

Step 2: Extract MFCC, LFPC, LPCC, ZCPA, PLP, R-PLP features from all speech segment training signals, where the number of Mel filters for MFCC and LFPC is 40; linear prediction for LPCC, PLP, and R-PLP The orders are 12, 16, and 16 respectively; the frequency segments of ZCPA are: 0, 106, 223, 352, 495, 655, 829, 1022, 1236, 1473, 1734, 2024, 2344, 2689, 3089, 3522, 4000 . Therefore, the dimensions of each type of feature of each statement are: ti*39, ti*40, ti*12, ti*16, ti*16, ti*16, where ti is the frame number of the i-th statement, and the multiplication sign The following numbers are the dimensions of each frame feature. In order to obtain the change of the speech signal in the time dimension, the first-order derivative and the second-order derivative are also calculated for the above-mentioned features in the time dimension. Finally, the dimensions of each type of feature are: ti*117, ti*140, ti*36, ti*48, ti*48, ti*48. The extracted speech signal feature of the ith sample is composed of all the above features, and the dimension is ti*(117+140+36+48+48+48).

Step 3: Use the following statistical functions: mean, standard deviation, minimum value (min), maximum value (max), kurtosis (kurtosis), and skewness (skewness) to obtain the above features in the time dimension Statistical results above. The feature statistics of labeled samples are recorded as {x ₁ , x ₂ ,...,x _n }, where n is the number of labeled samples.

Step 4: Divide {x ₁ ,x ₂ ,...,x _n } in the previous step into X _A =[x ₁ ,x ₂ ,...,x _m ] according to the label of the statement, X _B =[x _m+1 ,x _m+2 ,…,x _n ], where X _A is a segment of type A emotion, and X _B is a segment of type B emotion. The steps of training a sentence segment emotion classification method based on propensity cognitive learning are as follows:

(1) For x∈X _A , divide the angle between the sample in the Parzen window centered on x in X _A and the center sample x into multiple bins, and then use the following formula to calculate the distribution characteristics of the data around x in X _A .

b _x =[b ₁ ,b ₂ ,...,b _k ]

where b _j represents the jth box, 1(x _i ∈ X _j ) is 1 when x _i belongs to X _j , otherwise it is 0, X _j is a subset of X _A , and the difference between the samples in X _j and x is The angles between are distributed in the jth bin.

(2) For x∈X _A , divide the angle between the sample in the Parzen window centered on x in X _B and the center sample x into multiple bins, and then use the following formula to calculate the distribution characteristics of the data around x in X _A .

表示第

个箱子，1(x_i∈X_j)在x_i属于X_j时的值为1否则为0，X_j是X_B的子集，X_j内的样本与x之间的角度分布在第j个箱子中。in the formula

means the first

box, 1(x _i ∈ X _j ) is 1 when _xi belongs to X _j , otherwise 0, X _j is a subset of X _B , and the angle between the samples in X _j and x is distributed in the jth in a box.

(3) Use the following formula to calculate the difference in the data distribution of the two data sets near the x point:

表示两向量之间的距离，此处使用欧氏距离。in the formula

Represents the distance between two vectors, where Euclidean distance is used.

倾向于B情感的片段集合

以及倾向于中性情感的片段集合

其中

为d_x>T的x组成的集合。

为d_x<-T的x组成的集合。

为T>d_x>-T组成的集合。T是自主设置的阈值。对每个集合，再使用谱聚类的方法聚成多个区域，得到每个片段xi的区域标签

(4) According to the calculation result of the above step (1-4-3), the set of fragments tending to A emotion can be obtained

A collection of snippets leaning towards B sentiment

and a collection of clips that tend towards neutral emotions

in

is the set of x where d _x > T.

is the set of x where d _x <-T.

is a set consisting of T>d _x >-T. T is an autonomously set threshold. For each set, the spectral clustering method is used to cluster into multiple regions, and the region label of each segment xi is obtained.

L＝[L_A,L_B,L_C]，其中L_A∈R^p、L_B∈R^q、L_C∈R^u，p、q和u分别为

和

样本的个数，L_A、L_B和L_C中的元素值分别为1、2、3。使用下述目标方程，学习片段的特征子空间：(5) Definition

L=[L _A , L _B , L _C ], where L _A ∈ R ^p , L _B ∈ R ^q , L _C ∈ R ^u , p, q and u are respectively

and

The number of samples, the element values in L _A , L _B and L _C are 1, 2, and 3, respectively. The feature subspace of the segment is learned using the following objective equation:

J=J ₁ (o _i ,o _j )+β*J ₂ (o _i ,o _j )

和

三个类之间的类内距离较小，类间距离较大，定义如下：β is the equilibrium parameter. where J ₁ (o _i ,o _j ) can realize

and

The intra-class distance between the three classes is small, and the inter-class distance is large, which is defined as follows:

和

映射到子空间之后的结果。l_i和l_j对应o_i和o_j在L中的值。m是一个阈值，调整类间距离的范围，本发明中取1。G_ij为x_i和x_j之间的高斯距离。计算公式如下：where o _i and o _j are

and

The result after mapping to the subspace. l _i and l _j correspond to the values of o _i and o _j in L. m is a threshold, and the range of the distance between classes is adjusted, which is taken as 1 in the present invention. G _ij is the Gaussian distance between x _i and x _j . Calculated as follows:

J ₂ (o _i ,o _j ) can try to keep the relative relationship in each region unchanged, and the regions belonging to the same class are relatively close, but do not overlap. Defined as follows:

和

是xi和xj的区域标签。G_li是l_i类所有G_ij中的最大值。实现当两个片段属于同一区域时，保持他们之间的关系，当两者不属于同一区域但是属于同一类别时，以一个小的权重最小化他们之间的距离，可使两个区域尽量不重叠。in the formula

and

are the region labels for xi and xj. G _li is the maximum value _{among all G ij of} class li. Realize that when two fragments belong to the same area, keep the relationship between them, and when the two do not belong to the same area but belong to the same category, minimize the distance between them with a small weight, so that the two areas can be as different as possible. overlapping.

和

可得到W₁,W₂,…,W_q和b₁,b₂,…,b_q的值，

是求J对W的导数，

是求J对b的导数。In order to optimize the objective equation J, define o _i =φ(W _q φ(…W ₃ φ(W ₂ φ(W ₁ x _i +b ₁ )+b ₂ )+b ₃ )+b ₄ ), where φ( ·) is the sigmoid function, W ₁ , W ₂ ,...,W _q are the mapping matrices, and b ₁ ,b ₂ ,...,b _q are the offsets. by asking

and

The values of W ₁ , W ₂ ,...,W _q and b ₁ ,b ₂ ,...,b _q can be obtained,

is the derivative of J with respect to W,

is the derivative of J with respect to b.

和

的特征子空间，训练Sof tmax分类器将情感A，情感B和中性情感C分开。(6) for the above-mentioned steps (1-4-5) obtained

and

The feature subspace of , trains a Softmax classifier to separate sentiment A, sentiment B and neutral sentiment C.

(7) According to the operation process of the above step (1-4-5) and the above step (1-4-6), train a Softmax classifier capable of identifying all emotion pairs.

In this embodiment, the training voice samples of preset dimensions are obtained, and the test voice samples are segmented according to preset rules to obtain a plurality of initial training voice samples, and then feature extraction is performed on the initial training voice samples to obtain Features of the training voice signal to be processed, and perform feature statistics on the features of the training voice signal to be processed through a preset statistical function, obtain the statistical results of the training features to be confirmed, and perform preset multi-objective optimization according to the statistical results of the training features to be confirmed. The algorithm obtains target training feature data, then obtains the emotion category corresponding to the target training feature data according to the target training feature data, and establishes a preset Softmax classification model according to the emotion category and the target training feature data corresponding to the emotion category . Through the above method, the model can be trained for local segments of sentences, which can avoid different local segments in a sentence containing different emotions, or different local segments of the same emotion conflicting with each other, thereby reducing the difference between deep learning physical meaning and speech emotion recognition characteristics.

In addition, an embodiment of the present invention further provides a storage medium, where a speech emotion recognition program is stored, and when the speech emotion recognition program is executed by a processor, the steps of the speech emotion recognition method described above are implemented.

Referring to FIG. 4 , FIG. 4 is a structural block diagram of a first embodiment of a voice emotion recognition apparatus according to the present invention.

As shown in FIG. 4 , the voice emotion recognition device provided by the embodiment of the present invention includes: an acquisition module 4001, configured to acquire a test voice sample of a preset dimension, and perform segmentation processing on the test voice sample according to a preset rule to obtain a plurality of initial voice samples; the extraction module 4002 is used to extract the signal feature data of the initial voice samples to obtain the feature data of the to-be-processed voice signal; the statistics module 4003 is used to analyze the to-be-processed voice signal by a preset statistical function Feature statistics are performed on the feature data to obtain the feature statistics results to be confirmed; the calculation module 4004 is used to obtain feature target data through a preset multi-objective optimization algorithm according to the feature statistics results to be confirmed; the determination module 4005 is used to calculate the features The target data is input into the preset Softmax classification model to obtain speech emotion recognition results.

The obtaining module 4001 obtains a test voice sample of a preset dimension, and performs segmentation processing on the test voice sample according to a preset rule to obtain a plurality of initial voice samples.

It should be noted that, before the steps of obtaining a test voice sample of a preset dimension, and performing segmentation processing on the test voice sample according to a preset rule, and obtaining a plurality of initial voice samples, it is necessary to obtain a training voice sample of a preset dimension. , and perform segmentation processing on the test voice samples through preset rules to obtain a plurality of initial training voice samples, perform feature extraction on the initial training voice samples, and obtain the characteristics of the training voice signal to be processed. Perform feature statistics on the characteristics of the training voice signal to be processed, obtain the statistical results of the training characteristics to be confirmed, obtain target training characteristic data through a preset multi-objective optimization algorithm according to the statistical results of the training characteristics to be confirmed, and obtain the target training characteristic data according to the target training characteristic data Obtain the emotion category corresponding to the target training feature data, and establish a preset Softmax classification model according to the emotion category and the target training feature data corresponding to the emotion category.

In addition, it should be understood that the above-mentioned preset rules are user-defined sample division rules, that is to say, if the acquired test voice samples of the preset dimension have a corresponding duration of 5s, the preset rules are set as 0.2s, then 25 segments of initial speech samples of 0.2s are obtained after division according to preset rules.

In addition, it should be noted that the preset dimension mentioned above may be a time dimension or a non-time dimension, etc., which is not limited in this embodiment.

The extraction module 4002 performs an operation of extracting signal feature data on the initial speech sample to obtain the feature data of the speech signal to be processed.

In addition, it should be understood that, performing signal feature data extraction on the initial speech sample to extract Mel Frequency Cepstrum Coefficients (MFCC), Log Frequency Power Coefficients (LFPC), and Linear Prediction Cepstrum Coefficients (Linear Predictive Cepstral Coding, LPCC), Zero Crossing with Peak Amplitude (ZCPA), Perceptual Linear Predictive (Perceptual Linear Predictive, PLP), Rasta Filter Perceptual Linear Predictiv (Rasta Perceptual Linear Predictiv, R-PLP) ).

It should be understood that the feature extraction result of each type of feature mentioned above is a two-dimensional matrix, one of which is the time dimension, and then the first-order derivative ΔF _i and the second-order derivative of each type of feature F _i on the time dimension are calculated. ΔΔFi, and concatenate the original features, first-order derivative results, and second-order derivative results in the non-time dimension to form the final feature extraction result of each type of feature; the final feature extraction result of the above-mentioned features of all classes is in the non-time dimension. The above concatenation is the feature extraction result of the sample.

In addition, for ease of understanding, the following examples are provided:

Suppose, the corresponding F _MFCC ∈ R ^39×z , ΔF _MFCC ∈ R ^39×z , ΔΔF _i ∈ R ^39×z , where z is the number of frames, that is, the number of time dimensions, the concatenation result in the non-time dimension

串接后为

When MFCC and LPCC are connected, if

concatenated as

In addition, it should be understood that MFCC, LFPC, LPCC, ZCPA, PLP, R-PLP features are extracted during each speech signal feature extraction, wherein the number of Mel filters of MFCC and LFPC is 40; LPCC, PLP, The linear prediction orders of R-PLP are 12, 16, and 16 respectively; the frequency segments of ZCPA are: 0, 106, 223, 352, 495, 655, 829, 1022, 1236, 1473, 1734, 2024, 2344, 2689 , 3089, 3522, 4000. Therefore, the dimensions of each type of feature of each statement are: ti*39, ti*40, ti*12, ti*16, ti*16, ti*16, where ti is the frame number of the i-th statement, and the multiplication sign The following numbers are the dimensions of each frame feature. In order to obtain the change of the speech signal in the time dimension, the first-order derivative and the second-order derivative are also calculated for the above-mentioned features in the time dimension. Finally, the dimensions of each type of feature are: ti*117, ti*140, ti*36, ti*48, ti*48, ti*48. The extracted speech signal feature of the ith sample is composed of all the above features, and the dimension is ti*(117+140+36+48+48+48).

The statistics module 4003 performs feature statistics on the feature data of the to-be-processed voice signal through a preset statistical function, and obtains the feature statistics result to be confirmed.

It should be noted that the statistical functions are used to obtain the above features in the time dimension by using the mean, standard deviation, minimum value (min), maximum value (max), kurtosis (kurtosis) and skewness (skewness). Statistical results above.

In addition, it should be understood that the characteristic data of the label samples are obtained by screening from the statistical results obtained above, and the characteristic statistics of the label samples are carried out through a preset statistical function to obtain the statistical results of the characteristics to be confirmed, and there will be The feature statistics of labeled samples are recorded as {x ₁ , x ₂ ,...,x _n }, where n is the number of labeled samples.

The computing module 4004 is an operation of obtaining feature target data through a preset multi-objective optimization algorithm according to the statistical result of the feature to be confirmed.

In addition, it should be noted that {x ₁ ,x ₂ ,...,x _n } in the previous step is divided into X _A =[x ₁ ,x ₂ ,...,x _m ] according to the label of the statement, X _B =[x _m+1 ,x _m+2 ,...,x _n ], where X _A is a segment of type A emotion, and X _B is a segment of type B emotion, training a sentence segment emotion classification method based on propensity cognitive learning Proceed as follows:

(1) For x∈X _A , divide the angle between the sample in the Parzen window centered on x in X _A and the center sample x into multiple bins, and then use the following formula to calculate the distribution characteristics of the data around x in X _A .

b _x =[b ₁ ,b ₂ ,...,b _k ]

where b _j represents the jth box, 1(x _i ∈ X _j ) is 1 when x _i belongs to X _j , otherwise it is 0, X _j is a subset of X _A , and the difference between the samples in X _j and x is The angles between are distributed in the jth bin.

(2) For x∈X _A , divide the angle between the sample in the Parzen window centered on x in X _B and the center sample x into multiple bins, and then use the following formula to calculate the distribution characteristics of the data around x in X _A .

表示第

个箱子，1(x_i∈X_j)在x_i属于X_j时的值为1否则为0，X_j是X_B的子集，X_j内的样本与x之间的角度分布在第j个箱子中。in the formula

means the first

box, 1(x _i ∈ X _j ) is 1 when _xi belongs to X _j , otherwise 0, X _j is a subset of X _B , and the angle between the samples in X _j and x is distributed in the jth in a box.

(3) Use the following formula to calculate the difference in the data distribution of the two data sets near the x point:

表示两向量之间的距离，此处使用欧氏距离。in the formula

Represents the distance between two vectors, where Euclidean distance is used.

倾向于B情感的片段集合

以及倾向于中性情感的片段集合

其中

为d_x>T的x组成的集合。

为d_x<-T的x组成的集合。

为T>d_x>-T组成的集合。T是自主设置的阈值。对每个集合，再使用谱聚类的方法聚成多个区域，得到每个片段xi的区域标签

(4) According to the calculation result of the previous step, the set of fragments tending to A emotion can be obtained

A collection of snippets leaning towards B sentiment

and a collection of clips that tend towards neutral emotions

in

is the set of x where d _x > T.

is the set of x where d _x <-T.

is a set consisting of T>d _x >-T. T is an autonomously set threshold. For each set, the spectral clustering method is used to cluster into multiple regions, and the region label of each segment xi is obtained.

L＝[L_A,L_B,L_C]，其中L_A∈R^p、L_B∈R^q、L_C∈R^u，p、q和u分别为

和

样本的个数，L_A、L_B和L_C中的元素值分别为1、2、3。使用下述目标方程，学习片段的特征子空间：(5) Definition

L=[L _A , L _B , L _C ], where L _A ∈ R ^p , L _B ∈ R ^q , L _C ∈ R ^u , p, q and u are respectively

and

The number of samples, the element values in L _A , L _B and L _C are 1, 2, and 3, respectively. The feature subspace of the segment is learned using the following objective equation:

J=J ₁ (o _i ,o _j )+β*J ₂ (o _i ,o _j )

和

三个类之间的类内距离较小，类间距离较大，定义如下：β is the equilibrium parameter. where J ₁ (o _i ,o _j ) can realize

and

The intra-class distance between the three classes is small, and the inter-class distance is large, which is defined as follows:

和

映射到子空间之后的结果。l_i和l_j对应o_i和o_j在L中的值。m是一个阈值，调整类间距离的范围，本发明中取1。G_ij为x_i和x_j之间的高斯距离。计算公式如下：where o _i and o _j are

and

The result after mapping to the subspace. l _i and l _j correspond to the values of o _i and o _j in L. m is a threshold, and the range of the distance between classes is adjusted, which is taken as 1 in the present invention. G _ij is the Gaussian distance between x _i and x _j . Calculated as follows:

J ₂ (o _i ,o _j ) can try to keep the relative relationship in each region unchanged, and the regions belonging to the same class are relatively close, but do not overlap. Defined as follows:

和

是xi和xj的区域标签。G_li是l_i类所有G_ij中的最大值。实现当两个片段属于同一区域时，保持他们之间的关系，当两者不属于同一区域但是属于同一类别时，以一个小的权重最小化他们之间的距离，可使两个区域尽量不重叠。in the formula

and

are the region labels for xi and xj. G _li is the maximum value _{among all G ij of} class li. Realize that when two fragments belong to the same area, keep the relationship between them, and when the two do not belong to the same area but belong to the same category, minimize the distance between them with a small weight, so that the two areas can be as different as possible. overlapping.

和

可得到W₁,W₂,…,W_q和b₁,b₂,…,b_q的值，

是求J对W的导数，

是求J对b的导数。To optimize the objective equation J, we define o _i =φ(W _q φ(…W ₃ φ(W ₂ φ(W ₁ x _i +b ₁ )+b ₂ )+b ₃ )+b ₄ ) where φ( ·) is the sigmoid function, W ₁ , W ₂ ,...,W _q are the mapping matrices, and b ₁ ,b ₂ ,...,b _q are the offsets. by asking

and

The values of W ₁ , W ₂ ,...,W _q and b ₁ ,b ₂ ,...,b _q can be obtained,

is the derivative of J with respect to W,

is the derivative of J with respect to b.

The determining module 4005 inputs the feature target data into the preset Softmax classification model, and obtains the voice emotion recognition result.

In addition, it should be understood that, according to the above steps to obtain W ₁ , W ₂ ,...,W _q and b ₁ ,b ₂ ,...,b _q , calculate the features of {x ₁ ,x ₂ ,...,x _m } Select result z.

In addition, it should be noted that the above-mentioned W ₁ , W ₂ ,...,W _q and b ₁ ,b ₂ ,...,b _q are characteristic target data in this application.

Furthermore, it should be understood that the speech emotion categories { _{l 1 , l 2 ,} ..., x _m } are respectively obtained using the preset Softmax classifier obtained during the training process. l _m }. Then vote for the sentiment of the statement according to {l ₁ ,l ₂ ,...,l _m }.

In addition, it should be noted that the feature target data is input into the preset Softmax classification model to obtain voice emotion category data, and data statistics are performed on the voice emotion category data to obtain the voice emotion category data value. The speech emotion category data value is used to obtain speech emotion recognition results.

In addition, the above-mentioned step of obtaining the voice emotion recognition result according to the voice emotion category data value is to judge whether the voice emotion category data value belongs to the preset voice emotion category threshold range, and if the voice emotion category data value belongs to all If the preset voice emotion category threshold range is selected, the voice emotion recognition result is obtained according to the voice emotion category data value; if the voice emotion category data value does not belong to the preset voice emotion category threshold value range, return the The feature target data is input into the preset Softmax classification model to obtain the voice emotion category data.

In addition, it should also be noted that the corpus used in the effect evaluation of emotion recognition of the present invention is a standard database in the field of speech emotion recognition. The training process is completed first, followed by the recognition test. The test pattern is performed in a 5-fold crossover. It can identify 7 emotions of anger, fear, irritability, disgust, happiness, neutrality, and sadness. The average classification accuracy rate is 94.65% in the case of speaker dependence. Except for happiness and anger, which are more easily confused, the degree of discrimination between other emotions better. The average classification accuracy is 89.30% in the case of speaker independence.

It should be understood that the above are only examples, and do not constitute any limitation to the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as required, which is not limited by the present invention.

In this embodiment, a test voice sample of a preset dimension is obtained first, and a plurality of initial voice samples are obtained by segmenting the test voice sample according to a preset rule, and then signal feature data is extracted for the initial voice sample to obtain the voice to be processed. signal feature data, and filter the feature data of the to-be-processed voice signal to obtain label sample feature data, perform feature statistics on the label sample feature data through a preset statistical function, obtain feature statistics results to be confirmed, and then The statistical results of the training features to be confirmed are divided into emotion categories, and the training feature data to be optimized corresponding to different emotion categories is obtained. According to the training feature data to be optimized, the target training feature data is obtained through a preset multi-objective optimization algorithm, and finally the feature target data Input into the preset Softmax classification model to obtain speech emotion recognition results. Through the above method, it is possible to make full use of speech emotional fragments and the emotional relationship between sentences and fragments to form a kind of tendency data, so that the process of human processing tendency can be simulated, and the unbalanced information of the data can be used to compare with each other. Constraints to separate segments of different sentiments, thereby increasing sample size and increasing sample diversity.

It should be noted that the above-described workflow is only illustrative, and does not limit the protection scope of the present invention. In practical applications, those skilled in the art can select some or all of them to implement according to actual needs. The purpose of the solution in this embodiment is not limited here.

In addition, for technical details that are not described in detail in this embodiment, reference may be made to the speech emotion recognition method provided by any embodiment of the present invention, and details are not repeated here.

Furthermore, it should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements, but also other elements not expressly listed or inherent to such a process, method, article or system. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system that includes the element.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on such understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as a read-only memory). , ROM)/RAM, magnetic disk, optical disk), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in the various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields , are similarly included in the scope of patent protection of the present invention.

Claims (8)

1. A speech emotion recognition method, characterized in that the method comprises:

acquiring a test voice sample with a preset dimension, and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples;

extracting signal characteristic data of the initial voice sample to obtain voice signal characteristic data to be processed;

performing feature statistics on the voice signal feature data to be processed through a preset statistical function to obtain a feature statistical result to be confirmed;

obtaining feature target data through a preset multi-target optimization algorithm according to the feature statistical result to be confirmed;

inputting the characteristic target data into a preset Softmax classification model to obtain a speech emotion recognition result;

the method for obtaining the initial voice samples comprises the following steps of obtaining a test voice sample with a preset dimensionality, carrying out segmentation processing on the test voice sample through a preset rule, and obtaining a plurality of initial voice samples, wherein before the step of obtaining the initial voice samples, the method further comprises the following steps:

acquiring training voice samples with preset dimensions, and performing segmentation processing on the test voice samples through preset rules to obtain a plurality of initial training voice samples;

performing feature extraction on the initial training voice sample to obtain the features of a training voice signal to be processed;

performing feature statistics on the training voice signal features to be processed through a preset statistical function to obtain a training feature statistical result to be confirmed;

carrying out emotion category division on the statistical result of the training features to be confirmed to obtain training feature data to be optimized corresponding to different emotion categories;

determining a peripheral data distribution characteristic set corresponding to each training characteristic data to be optimized;

determining data distribution differences corresponding to different emotion classes according to the surrounding data distribution feature set;

obtaining emotion fragment sets corresponding to different emotion types according to the data distribution difference;

determining a feature subspace corresponding to each emotion fragment set;

and establishing a preset Softmax classification model based on the plurality of feature subspaces.

2. The method of claim 1, wherein the step of inputting the feature target data into a preset Softmax classification model to obtain the speech emotion recognition result comprises:

inputting the characteristic target data into the preset Softmax classification model to obtain speech emotion category data;

performing data statistics on the voice emotion type data to obtain a voice emotion type data value;

and obtaining a voice emotion recognition result according to the voice emotion category data value.

3. The method of claim 2, wherein the step of obtaining speech emotion recognition results according to the speech emotion classification data values comprises:

judging whether the voice emotion type data value belongs to a preset voice emotion type threshold range or not;

and if the voice emotion type data value belongs to the preset voice emotion type threshold range, acquiring a voice emotion recognition result according to the voice emotion type data value.

4. The method of claim 3, wherein the step of determining whether the speech emotion classification data value falls within a preset speech emotion classification threshold range further comprises:

and if the voice emotion category data value does not belong to the preset voice emotion category threshold range, returning to the step of inputting the feature target data into the preset Softmax classification model to obtain voice emotion category data.

5. The method according to claim 1, wherein the step of performing feature statistics on the feature data of the speech signal to be processed by using a preset statistical function to obtain a feature statistical result to be confirmed comprises:

screening the voice signal characteristic data to be processed to obtain label sample characteristic data;

and carrying out feature statistics on the tag sample feature data through a preset statistical function to obtain a feature statistical result to be confirmed.

6. An apparatus for speech emotion recognition, the apparatus comprising:

the system comprises an acquisition module, a processing module and a control module, wherein the acquisition module is used for acquiring a test voice sample with a preset dimensionality and performing segmentation processing on the test voice sample through a preset rule to obtain a plurality of initial voice samples;

the extraction module is used for extracting signal characteristic data of the initial voice sample to obtain voice signal characteristic data to be processed;

the statistical module is used for carrying out feature statistics on the voice signal feature data to be processed through a preset statistical function to obtain a feature statistical result to be confirmed;

the calculation module is used for obtaining feature target data through a preset multi-target optimization algorithm according to the feature statistical result to be confirmed;

the determining module is used for inputting the characteristic target data into a preset Softmax classification model to obtain a speech emotion recognition result;

the speech emotion recognition apparatus further includes: acquiring training voice samples with preset dimensions, and performing segmentation processing on the test voice samples through preset rules to obtain a plurality of initial training voice samples; performing feature extraction on the initial training voice sample to obtain the features of a training voice signal to be processed; performing feature statistics on the training voice signal features to be processed through a preset statistical function to obtain a training feature statistical result to be confirmed; carrying out emotion category division on the statistical result of the training features to be confirmed to obtain training feature data to be optimized corresponding to different emotion categories; determining a peripheral data distribution characteristic set corresponding to each training characteristic data to be optimized; determining data distribution differences corresponding to different emotion classes according to the surrounding data distribution feature set; obtaining emotion fragment sets corresponding to different emotion types according to the data distribution difference; determining a feature subspace corresponding to each emotion fragment set; and establishing a preset Softmax classification model based on the plurality of feature subspaces.

7. An electronic device, characterized in that the device comprises: a memory, a processor and a speech emotion recognition program stored on the memory and executable on the processor, the speech emotion recognition program being configured to implement the steps of the speech emotion recognition method as claimed in any of claims 1 to 5.

8. A storage medium having stored thereon a speech emotion recognition program, which when executed by a processor implements the steps of the speech emotion recognition method as claimed in any one of claims 1 to 5.

Images