CN117708754A - A multimodal emotion recognition method and device - Google Patents

Abstract

The present invention provides a multi-modal emotion recognition method and device, which includes: preprocessing original multi-modal data samples, obtaining primary features of each single modality and inputting them into the single-modal feature extraction network respectively, and extracting each single modal data sample. The high-level features of the modality, after one-dimensionalization, are input into each single-modal recognition network respectively, and the single-modal model composed of each single-modal feature extraction network and the single-modal recognition network is trained; each single-modal recognition network is obtained. The weight of the modal feature extraction network is extracted, and the activation degree of all neurons in each single-modal recognition network is extracted; its type is judged based on the activation degree of each neuron, and the unique connection constraints between different neurons are obtained; based on The obtained weight of each single-modal feature extraction network and the extracted connection constraints are used to establish a complete multi-modal emotion recognition model and complete the training, which is used to identify multi-modal emotions. The invention realizes cross-modal interaction and can effectively identify multi-modal emotions.

Description

A multimodal emotion recognition method and device

Technical field

The present invention relates to the technical field of multi-modal recognition, and in particular to a multi-modal emotion recognition method and device.

Background technique

Multimodal recognition can effectively utilize information from multiple different modalities. These information complement each other and can effectively alleviate the problems of noisy data, non-universality and other problems faced by single-modal recognition. Therefore, it can be used in various multimodal recognition tasks. (such as brain-inspired multi-modal emotion recognition) has achieved extensive progress and good performance.

For multimodal recognition, how to effectively utilize information from multiple modalities is a key issue. In order to solve this problem, multi-modal fusion is a key means. With the development of deep learning, more and more methods are used to fuse multi-modal information, such as data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion means combining input data from different modalities at the data level to merge it into more information-rich data, and then extract features from the combined data and build a classification model to identify and classify the extracted features; feature-level fusion refers to Create a combined feature from multiple input features extracted from different data modalities, and then use a classifier to identify the combined features; in decision-level fusion, a separate classification system classifies data from each modality into relevant categories, and then combine the scores of each classification system to obtain the final recognition result. However, most of these deep learning methods do not take into account the interaction between different modalities, which is called cross-modal interaction. Cross-modal interaction can make the information of each modality influence each other and complement each other, which is crucial for effective multi-modal fusion. Therefore, traditional deep learning methods have low recognition accuracy in multi-modal recognition tasks due to ignoring cross-modal interaction. Still needs to be improved.

Contents of the invention

The present invention provides a multi-modal emotion recognition method and device to solve the problems existing in the existing technology. The technical solutions provided by the present invention are as follows:

On the one hand, a multi-modal emotion recognition method is provided, and the method includes:

S1. Preprocess the original multi-modal data sample and obtain the primary features of each single modality included in the original multi-modal data sample;

S2. Input the primary features of each single modality into the single-modal feature extraction network, and extract the advanced features of each single modality;

S3. After making the high-level features of each single modality one-dimensional, input each single-modal recognition network separately, and perform a single-modal model composed of each single-modal feature extraction network and single-modal recognition network. conduct training;

S4. Obtain the weight of each single-modal feature extraction network from the trained single-modal model, and extract the activation degree of all neurons in each single-modal recognition network;

S5. Determine the type of each neuron based on its activation degree, and obtain unique connection constraints between different neurons based on the type of each neuron;

S6. Based on the obtained weight of each single-modal feature extraction network and the extracted connection constraints, establish a complete multi-modal emotion recognition model and conduct training, and obtain the trained multi-modal emotion recognition model, which can be used to identify multiple Modal emotions.

Optionally, the S1 specifically includes:

The original multi-modal data sample is a video clip with sound. Several frames of pictures are extracted at equal intervals. The facial contour of the person is extracted from each frame of picture as the primary feature of the visual modality. Use express;

Extract the Mel spectral coefficient features of the audio of the video clip as the primary features of the auditory modality, using express.

Optionally, each single-modal feature extraction network in S2 is a spiking neural network using the classic LIF model as the neuron model. Each network is composed of a convolution layer and a pooling layer, and is used to extract from The primary features of the visual modality and the auditory modality further extract the high-level features of the visual modality or the auditory modality. The high-level features of these two modalities are respectively used. with/> express.

Optionally, each single-modal recognition network in S3 is a spiking neural network using the classic LIF model as the neuron model. Each network is composed of two fully connected layers and one output layer, for Identify the emotions expressed in each single-modal data;

The neuron model of each single-modal recognition network adds an adaptive activation threshold, and the threshold is set by an ordinary differential equation;

Among them, in the first fully connected layer in the visual modality recognition network, the membrane potential of the i-th neuron is updated as follows:

In the first fully connected layer in the auditory modality recognition network, the membrane potential of the i-th neuron is updated as follows:

Among them, C represents the capacitance, g represents the conductance, V _i (t) represents the membrane potential of the i-th neuron at time t, _Si (t) represents the activation flag of this neuron, V ₁ represents the resting potential, V ₂ represents the reset potential, V _th represents the activation threshold, Represents the weight of the j-th neuron in the high-level feature of the input visual modality or auditory modality to the i-th neuron in the first layer of fully connected network. The dynamic threshold a _i (t) is the discharge after resetting to the membrane potential. Accumulated during the period, as the discharge frequency increases, the threshold value also increases, and as the discharge frequency decreases, the threshold value also decreases. α, β, and γ are all hyperparameters.

Optionally, extracting the activation degree of all neurons in each single-modal recognition network in S4 specifically includes:

In each single-modal recognition network, for each sample, record the membrane potential of all spiking neurons in the l-th layer at all times, using represents, the membrane potential matrix is used to represent the activation degree of the neuron, where i represents the i-th sample and l represents the l-th layer.

Optionally, in S5, the type of each neuron is determined based on its activation degree, specifically including:

By averaging the activation of neurons over time scales and samples, the spike pattern is obtained, denoted as The formula is:

Among them, Mean _t means calculating the average on the time scale, Means averaging over samples;

according to Obtain multi-sensory neurons, represented by the set M _l :

Among them, Top means to find the neuron with the strongest spike level according to the spike pattern of the neuron, ρ is a hyperparameter, indicating that the neuron with the top activation degree ρ is selected as the neuron with the strongest discharge level. The lth layer Multisensory neurons, denoted by M _l , are the intersection of the most strongly activated visual and auditory neurons;

Unisensory neurons are the difference set of all neurons in layer l and multisensory neurons:

in, Represents all neurons in layer l, /> Represent the unisensory neurons of layer l respectively.

Optionally, in S5, unique connection constraints between different neurons are obtained according to the type of each neuron, specifically including:

The connection weight W ₁ between the high-level features of each single modality and the fully connected neurons in the first layer of each single mode recognition network, and the connection weight W between the fully connected neurons in the first and second layers _2. Establish connection constraints for these two weights according to the following rules: mask ¹ , mask ² , mask ¹ , and mask ² are matrices filled with 1 or 0, where 1 and 0 respectively indicate whether the connection between neurons exists, according to The following rules determine the connection weight of i→j:

a) If neuron i represents a single-sensory neuron, and neuron j represents a single-sensory neuron of the same modality, then the connection weight of i→j exists,

b) If neuron i represents a single-sensory neuron, and neuron j represents a multi-sensory neuron, then the connection weight of i→j exists,

c) If neuron i represents a multi-sensory neuron and neuron j represents a single-sensory neuron, then the connection weight of i→j exists,

In other cases, the connection weight of i→j does not exist,

Optionally, the complete multi-modal emotion recognition model in S6 includes each single-modal feature extraction network and multi-modal recognition network;

The multi-modal recognition network has a similar architecture to each single-modal recognition network, including two fully connected layers and one output layer. The difference is:

a) Each fully connected layer neuron in the single-modal recognition network has a fixed number of neurons, but the number of neurons in the fully connected layer in the multi-modal recognition network is The change is the sum of the numbers of the two types of single-sensory neurons and multi-sensory neurons extracted. The calculation formula is:

b) The layers in the multi-modal recognition network are sparsely connected, and the extracted connection constraints mask ¹ and mask ² are used, between the high-level features of each single modality and the neurons of the first fully connected layer The connection weight W ₁ and the connection weight W ₂ between the fully connected neurons in the first layer and the second layer, calculate the Hadamard product of mask ¹ and mask ² with W ₁ and W ₂ respectively, and calculate the Hadamard product between the layers. The connection is subject to regular restrictions, and the calculation formula is:

Among them, * represents the Hadamard product, Represents the element of the i-th row and j-th column in the matrix of connection constraint mask ¹ , W _{1, i, j} represents the element of the i-th row and j-th column of the matrix of connection weight W ₁ , /> Represents the element in the i-th row and j-th column in the matrix of connection constraint mask ² , and W _{2, i, j} represents the element in the i-th row and j-th column in the matrix of connection weight W ₂ .

Optionally, a complete multi-modal emotion recognition model is trained in S6, including:

The original multi-modal data samples are subjected to single-modal data preprocessing and primary feature extraction. The single-modal primary features are input into the single-modal feature extraction network respectively to obtain single-modal high-level features. The single-modal high-level features are one-dimensionally After transformation, the spliced features are input to the multi-modal recognition network, and the entire model is trained using the DFA algorithm. During the training process, the weight of each single-modal feature extraction network is fixed, which is the weight after training in S3. Only the weights in the multimodal recognition network are trained.

On the other hand, a multi-modal emotion recognition device is provided, and the device includes:

The first acquisition module is used to preprocess the original multi-modal data sample and obtain the primary features of each single modality included in the original multi-modal data sample;

The first extraction module is used to input the primary features of each single modality into the single-modality feature extraction network respectively, and extract the high-level features of each single modality;

The first training module is used to convert the high-level features of each single modality into one dimension and then input them into each single-modality recognition network respectively. The pair consists of each single-modality feature extraction network and a single-modality recognition network. Single-modal model for training;

The second acquisition module is used to obtain the weight of each single-modal feature extraction network from the trained single-modal model, and extract the activation degree of all neurons in each single-modal recognition network;

The third acquisition module is used to determine the type of each neuron based on its activation degree, and obtain unique connection constraints between different neurons based on the type of each neuron;

The second training module is used to establish a complete multi-modal emotion recognition model and train it based on the obtained weight of each single-modal feature extraction network and the extracted connection constraints, and obtain the trained multi-modal emotion recognition model. , for identifying multimodal emotions.

On the other hand, an electronic device is provided. The electronic device includes a processor and a memory. Instructions are stored in the memory. The instructions are loaded and executed by the processor to implement the above multi-modal emotion recognition method.

On the other hand, a computer-readable storage medium is provided, in which instructions are stored, and the instructions are loaded and executed by a processor to implement the above multi-modal emotion recognition method.

Compared with the existing technology, the above technical solution has at least the following beneficial effects:

Inspired by the theory of neuron diversity in the human brain's process of perceiving multi-modal information, the invention realizes cross-modal interaction and has higher multi-modal emotion recognition accuracy than existing brain-like methods.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of a multi-modal emotion recognition method provided by an embodiment of the present invention;

Figure 2 is a flow chart of a single-modal model recognition method using the auditory modality as an example provided by an embodiment of the present invention;

Figure 3 is a schematic diagram of determining the type of each neuron in a single-modal recognition network based on its activation degree according to an embodiment of the present invention;

Figure 4 is a schematic diagram of designing unique connection constraints between neurons according to the type of each neuron and applying it to a multi-modal recognition network provided by an embodiment of the present invention;

Figure 5 is a flow chart of a method for identifying multi-modal emotions to be recognized according to an embodiment of the present invention;

Figure 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the described embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

As shown in Figure 1, an embodiment of the present invention provides a multi-modal emotion recognition method, which includes:

S1. Preprocess the original multi-modal data sample and obtain the primary features of each single modality included in the original multi-modal data sample;

S2. Input the primary features of each single modality into the single-modal feature extraction network, and extract the advanced features of each single modality;

S3. After making the high-level features of each single modality one-dimensional, input each single-modal recognition network separately, and perform a single-modal model composed of each single-modal feature extraction network and single-modal recognition network. conduct training;

S4. Obtain the weight of each single-modal feature extraction network from the trained single-modal model, and extract the activation degree of all neurons in each single-modal recognition network;

S5. Determine the type of each neuron based on its activation degree, and obtain unique connection constraints between different neurons based on the type of each neuron;

S6. Based on the obtained weight of each single-modal feature extraction network and the extracted connection constraints, establish a complete multi-modal emotion recognition model and conduct training, and obtain the trained multi-modal emotion recognition model, which can be used to identify multiple Modal emotions.

The human brain can interact with information from multiple modalities such as vision and hearing to achieve accurate and efficient perception and recognition. Neuroscience research shows that this ability of the brain originates from a variety of neurons, including unisensory neurons and multisensory neurons. A characteristic feature of these neurons is that they respond differently to different sensory stimuli. Specifically, unisensory neurons respond only to a single sensory stimulus, while multisensory neurons respond to stimuli from multiple senses. Since these neurons have different response characteristics, the human brain can effectively capture cross-modal interactions, resulting in superior multi-modal recognition capabilities. By drawing on the neuron diversity behind the advantages of the human brain in multi-modal recognition, the embodiment of the present invention proposes a multi-modal emotion recognition method inspired by neuron diversity. The following is detailed in conjunction with Figures 2-5. Describe a multi-modal emotion recognition method provided by embodiments of the present invention, including:

S1. Preprocess the original multi-modal data sample and obtain the primary features of each single modality included in the original multi-modal data sample;

Optionally, the S1 specifically includes:

The original multi-modal data sample is a video clip with sound. Several frames of pictures (such as 15 frames) are extracted at equal intervals, and a person's facial contour is extracted from each frame of picture (the picture can be compressed to 28 *28 size), as the primary feature of the visual modality, use express;

The dimension of each frame in the embodiment of the present invention is 784, so the final dimension of the initial feature of the visual modality of each sample is

Extract the Mel-scale FrequencyCepstral Coefficients (MFCC) features of the audio of the video clip as the primary features of the auditory modality, using express.

The average number of frames of all speech sounds in the embodiment of the present invention is 280 frames, and the feature dimension of each frame is 12. Therefore, the initial features of the auditory modality of each sample The final dimension of is/>

The specific extraction methods of primary features of the visual modality and the extraction methods of primary features of the auditory modality are existing technologies and will not be described in detail here.

The embodiment of the present invention takes visual modality and auditory modality as examples to illustrate the method of the embodiment of the present invention. However, the multi-modal data in the embodiment of the present invention may also include data in other modalities such as text. The embodiment of the present invention is suitable for multi-modal data. There is no limitation on the status data, and they are all within the protection scope of the embodiments of the present invention.

S2. Input the primary features of each single modality into the single-modal feature extraction network, and extract the advanced features of each single modality;

Optionally, each single-modality feature extraction network (visual modality extraction network and auditory modality extraction network) in S2 is a spiking neural network (SNN) using the classic LIF model as a neuron model. ), each network consists of a convolutional layer and a pooling layer, used to further extract high-level features of the visual modality or auditory modality from the primary features of the visual modality and auditory modality respectively. The two modalities Advanced features are used separately with/> express.

S3. After converting the high-level features of each single modality into one dimension (specifically, Flatten operation can be used for one-dimensionalization), input each single-modality recognition network (visual modality recognition network and auditory modality recognition network) respectively. ), train the single-modal model composed of each single-modal feature extraction network and single-modal recognition network;

Optionally, each single-modal recognition network in S3 is a spiking neural network using the classic LIF model as the neuron model. Each network is composed of two fully connected layers and one output layer, for Identify the emotions expressed in each single-modal data;

Each fully connected layer in each single-modal recognition network in the embodiment of the present invention has a fixed number (such as 200) of neurons, and the number of neurons in the output layer is consistent with the number of emotion labels in the training data set.

The embodiment of the present invention considers the plasticity of neurons in the single-modal recognition network. The neuron model of each single-modal recognition network adds an adaptive activation threshold, and the threshold is set by an ordinary differential equation;

Among them, in the first fully connected layer in the visual modality recognition network, the membrane potential of the i-th neuron is updated as follows:

In the first fully connected layer in the auditory modality recognition network, the membrane potential of the i-th neuron is updated as follows:

Among them, C represents the capacitance, g represents the conductance, V _i (t) represents the membrane potential of the i-th neuron at time t, _Si (t) represents the activation flag of this neuron, V ₁ represents the resting potential, V ₂ represents the reset potential, V _th represents the activation threshold, Represents the weight of the j-th neuron in the high-level feature of the input visual modality or auditory modality to the i-th neuron in the first layer of fully connected network. The dynamic threshold a _i (t) is the discharge after resetting to the membrane potential. accumulated during the period. As the discharge frequency increases, the threshold value also increases. As the discharge frequency decreases, the threshold value also decreases. α, β, and γ are all hyperparameters (without practical significance. In the embodiment of the present invention, α =0.9, β=0.1, γ=1);

For the single-modal model, based on the samples in the training set and their emotional labels (the manually labeled emotional categories to which the samples belong), Direct Feedback Alignment (DFA) is used for supervised training to obtain the single-modal model that has been trained. Model.

The identification method of a single-modal model (taking the auditory modality as an example) according to the embodiment of the present invention is shown in Figure 2.

S4. Obtain the weight of each single-modal feature extraction network from the trained single-modal model, and extract the activation degree of all neurons in each single-modal recognition network;

For the trained single-modal model, save the weight of each single-modal feature extraction network used to extract high-level features of the visual modality and auditory modality, represented by En ^(v) and En ^(a) respectively.

Optionally, extracting the activation degree of all neurons in each single-modal recognition network in S4 specifically includes:

In each single-modal recognition network, for each sample, record the membrane potential of all spiking neurons in the l-th layer at all times, using represents, the membrane potential matrix is used to represent the activation degree of the neuron, where i represents the i-th sample and l represents the l-th layer.

S5. Determine the type of each neuron based on its activation degree, and obtain unique connection constraints between different neurons based on the type of each neuron;

The embodiment of the present invention is inspired by the characteristics of single-sensory neurons and multi-sensory neurons in the human brain that have different responses to different stimuli. The type of each neuron in the single-modal recognition network is determined based on its activation degree. Specifically:

The response characteristics of neuronal diversity in the human brain are as follows:

In the human brain, multisensory information can be efficiently processed and recognized, depending on different neurons that exhibit different responses to sensory input. In areas of the human brain responsible for integrating information from multiple modalities, a variety of neurons, including multisensory neurons and unisensory neurons, are ubiquitous. Multisensory neurons are defined as neurons that respond to stimulation from more than one sense, while unisensory neurons are those that respond to stimulation from only one sense. Unisensory neurons can be further divided into visual and auditory unisensory neurons. For multi-sensory neurons, when the external stimuli it receives have a common source, its response significantly exceeds the response to any single sensory stimulus, while the response of unisensory neurons to multi-sensory stimuli is different from that of single-sensory neurons. There were no significant differences in responses to stimulation.

Optionally, as shown in Figure 3, the type of each neuron is determined according to its activation degree in S5, specifically including:

By averaging the activation of neurons over time scales and samples, the spike pattern is obtained, denoted as The formula is:

Among them, Mean _t means calculating the average on the time scale, Means averaging over samples;

according to Obtain multi-sensory neurons, represented by the set M _l :

Among them, Top means to find the neuron with the strongest spike level based on the spike pattern of the neuron, and ρ is a hyperparameter (the value range is 0 to 1), indicating that the neuron with the highest activation degree ρ is selected as the neuron with the strongest discharge. Horizontal neurons (for example, if ρ has a value of 0.5, then select the top 50% of neurons), the multi-sensory neurons of layer l, represented by M _l , are the visual neurons and auditory neurons with the strongest activation intersection;

Unisensory neurons are the difference set of all neurons in layer l and multisensory neurons:

in, Represents all neurons in layer l, /> Represent the unisensory neurons of layer l respectively.

Optionally, in S5, according to the type of each neuron, unique connection constraints between different neurons are obtained (inspired by the response characteristics of single-sensory neurons and multi-sensory neurons in the human brain), specifically including:

The connection weight W ₁ between the high-level features of each single modality and the fully connected neurons in the first layer of each single mode recognition network, and the connection weight W between the fully connected neurons in the first and second layers _2. Establish connection constraints for these two weights according to the following rules: mask ¹ , mask ² , mask ¹ , and mask ² are matrices filled with 1 or 0, where 1 and 0 respectively indicate whether the connection between neurons exists, according to The following rules determine the connection weight of i→j:

a) If neuron i represents a single-sensory neuron (visual modality or auditory modality), and neuron j represents a single-sensory neuron of the same modality, then the connection weight of i→j exists,

b) If neuron i represents a single-sensory neuron (visual modality or auditory modality), and neuron j represents a multi-sensory neuron, then the connection weight of i→j exists,

c) If neuron i represents a multi-sensory neuron, and neuron j represents a single-sensory neuron (visual modality or auditory modality), then the connection weight of i→j exists,

In other cases, the connection weight of i→j does not exist,

Through these constraints, cross-modal interaction is achieved. For example, auditory features Connect to the multi-sensory neuron M ₁ in the first fully connected layer, and then connect to the visual neuron in the second fully connected layer/> One-way transmission from hearing to vision is achieved. The same goes for one-way transmission from vision to hearing, thus enabling cross-modal interaction.

S6. Based on the obtained weight of each single-modal feature extraction network and the extracted connection constraints, establish a complete multi-modal emotion recognition model and conduct training, and obtain the trained multi-modal emotion recognition model, which can be used to identify multiple Modal emotions.

Optionally, the complete multi-modal emotion recognition model in S6 includes each single-modal feature extraction network and multi-modal recognition network;

The multi-modal recognition network has a similar architecture to each single-modal recognition network, including two fully connected layers and one output layer. The difference is:

a) Each fully connected layer neuron in the single-modal recognition network has a fixed number (such as 200) of neurons, but the number of neurons in the fully connected layer in the multi-modal recognition network The change is the sum of the numbers of the two types of single-sensory neurons and multi-sensory neurons extracted. The calculation formula is:

b) The layers in the multi-modal recognition network are sparsely connected, and the extracted connection constraints mask ¹ and mask ² are used, between the high-level features of each single modality and the neurons of the first fully connected layer The connection weight W ₁ and the connection weight W ₂ between the fully connected neurons in the first layer and the second layer, calculate the Hadamard product of mask ¹ and mask ² with W ₁ and W ₂ respectively, and calculate the Hadamard product between the layers. The connection is subject to regular restrictions, and the calculation formula is:

Among them, * represents the Hadamard product, Represents the element of the i-th row and j-th column in the matrix of connection constraint mask ¹ , W _{1, i, j} represents the element of the i-th row and j-th column of the matrix of connection weight W ₁ , /> Represents the element in the i-th row and j-th column in the matrix of connection constraint mask ² , and W _{2, i, j} represents the element in the i-th row and j-th column in the matrix of connection weight W ₂ .

The embodiment of the present invention designs unique connection constraints between neurons according to the type of each neuron and applies it to the multi-modal recognition network, as shown in Figure 4.

Optionally, a complete multi-modal emotion recognition model is trained in S6, including:

The original multi-modal data samples are subjected to single-modal data preprocessing and primary feature extraction. The single-modal primary features are input into the single-modal feature extraction network to obtain single-modal high-level features. The single-modal high-level features are one-dimensionally After transformation, the spliced features are input to the multi-modal recognition network, and the entire model is trained using the DFA algorithm. During the training process, the weight of each single-modal feature extraction network is fixed, which is the weight after training in S3. Only the weights in the multimodal recognition network are trained.

As shown in Figure 5, when using the method of the embodiment of the present invention to identify multi-modal emotions to be recognized, it includes:

Preprocess the original multi-modal data to be identified and obtain the primary features of each single modality (visual modality and auditory modality) included in the original multi-modal data sample to be identified;

Input the primary features of each single modality into the single-modality feature extraction network respectively, and extract the high-level features of each single modality;

The high-level features of each single modality are one-dimensionally spliced and then input into the multi-modal recognition network to identify the multi-modal emotions expressed by the original multi-modal data to be recognized.

An embodiment of the present invention also provides a multi-modal emotion recognition device, which includes:

The first acquisition module is used to preprocess the original multi-modal data sample and obtain the primary features of each single modality included in the original multi-modal data sample;

The first extraction module is used to input the primary features of each single modality into the single-modality feature extraction network respectively, and extract the high-level features of each single modality;

The first training module is used to convert the high-level features of each single modality into one dimension and then input them into each single-modality recognition network respectively. The pair consists of each single-modality feature extraction network and a single-modality recognition network. Single-modal model for training;

The second acquisition module is used to obtain the weight of each single-modal feature extraction network from the trained single-modal model, and extract the activation degree of all neurons in each single-modal recognition network;

The third acquisition module is used to determine the type of each neuron based on its activation degree, and obtain unique connection constraints between different neurons based on the type of each neuron;

The second training module is used to establish a complete multi-modal emotion recognition model and train it based on the obtained weight of each single-modal feature extraction network and the extracted connection constraints, and obtain the trained multi-modal emotion recognition model. , for identifying multimodal emotions.

The functional structure of a multi-modal emotion recognition device provided by an embodiment of the present invention corresponds to a multi-modal emotion recognition method provided by an embodiment of the present invention, and will not be described again here.

FIG. 6 is a schematic structural diagram of an electronic device 600 provided by an embodiment of the present invention. The electronic device 600 may vary greatly due to different configurations or performance, and may include one or more processors (central processing units, CPUs) 601 and one or more memories 602, wherein instructions are stored in the memory 602, and the instructions are loaded and executed by the processor 601 to implement the steps of the above multi-modal emotion recognition method.

In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the instructions can be executed by a processor in the terminal to complete the above-mentioned multi-modal emotion recognition method. For example, the computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage media mentioned can be read-only memory, magnetic disks or optical disks, etc.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (10)

1. A multi-modal emotion recognition method, characterized in that the method includes: S1. Preprocess the original multi-modal data sample and obtain the primary features of each single modality included in the original multi-modal data sample; S2. Input the primary features of each single modality into the single-modal feature extraction network, and extract the advanced features of each single modality; S3. After making the high-level features of each single modality one-dimensional, input each single-modal recognition network separately, and perform a single-modal model composed of each single-modal feature extraction network and single-modal recognition network. conduct training; S4. Obtain the weight of each single-modal feature extraction network from the trained single-modal model, and extract the activation degree of all neurons in each single-modal recognition network; S5. Determine the type of each neuron based on its activation degree, and obtain unique connection constraints between different neurons based on the type of each neuron; S6. Based on the obtained weight of each single-modal feature extraction network and the extracted connection constraints, establish a complete multi-modal emotion recognition model and conduct training, and obtain the trained multi-modal emotion recognition model, which can be used to identify multiple Modal emotions. 2. The method according to claim 1, characterized in that, S1 specifically includes: The original multi-modal data sample is a video clip with sound. Several frames of pictures are extracted at equal intervals. The facial contour of the person is extracted from each frame of picture as the primary feature of the visual modality. Use express; Extract the Mel spectral coefficient features of the audio of the video clip as the primary features of the auditory modality, using express. 3. The method according to claim 2, characterized in that each single-modal feature extraction network in the S2 is a spiking neural network using the classic LIF model as a neuron model, and each network is composed of a convolution layer and a pooling layer, used to further extract the high-level features of the visual modality or the auditory modality from the primary features of the visual modality and the auditory modality respectively. The high-level features of the two modalities are respectively used with/> express. 4. The method according to claim 3, characterized in that each single-modal recognition network in the S3 is a spiking neural network using the classic LIF model as a neuron model, and each network is composed of two layers of full-scale neural networks. The connection layer is composed of an output layer and is used to identify the emotions expressed in each single-modal data; The neuron model of each single-modal recognition network adds an adaptive activation threshold, and the threshold is set by an ordinary differential equation; Among them, in the first fully connected layer in the visual modality recognition network, the membrane potential of the i-th neuron is updated as follows: In the first fully connected layer in the auditory modality recognition network, the membrane potential of the i-th neuron is updated as follows: Among them, C represents the capacitance, g represents the conductance, V _i (t) represents the membrane potential of the i-th neuron at time t, _Si (t) represents the activation flag of this neuron, V ₁ represents the resting potential, V ₂ represents the reset potential, V _th represents the activation threshold, Represents the weight of the j-th neuron in the high-level feature of the input visual modality or auditory modality to the i-th neuron in the first layer of fully connected network. The dynamic threshold a _i (t) is the discharge after resetting to the membrane potential. Accumulated during the period, as the discharge frequency increases, the threshold value also increases, and as the discharge frequency decreases, the threshold value also decreases. α, β, and γ are all hyperparameters. 5. The method according to claim 1, characterized in that extracting the activation degree of all neurons in each single-modal recognition network in S4 specifically includes: In each single-modal recognition network, for each sample, record the membrane potential of all spiking neurons in the l-th layer at all times, using represents, the membrane potential matrix is used to represent the activation degree of the neuron, where i represents the i-th sample and l represents the l-th layer. 6. The method according to claim 5, characterized in that, in S5, judging the type of each neuron according to its activation degree specifically includes: By averaging the activation of neurons over time scales and samples, the spike pattern is obtained, denoted as The formula is: Among them, Mean _t means calculating the average on the time scale, Means averaging over samples; according to Obtain multi-sensory neurons, represented by the set M _l : Among them, Top means to find the neuron with the strongest spike level according to the spike pattern of the neuron, ρ is a hyperparameter, indicating that the neuron with the top activation degree ρ is selected as the neuron with the strongest discharge level. The lth layer Multisensory neurons, denoted by M _l , are the intersection of the most strongly activated visual and auditory neurons; Unisensory neurons are the difference set of all neurons in layer l and multisensory neurons: in, Represents all neurons in layer l, /> Represent the unisensory neurons of layer l respectively. 7. The method according to claim 6, characterized in that, in S5, unique connection constraints between different neurons are obtained according to the type of each neuron, specifically including: The connection weight W ₁ between the high-level features of each single modality and the fully connected neurons in the first layer of each single mode recognition network, and the connection weight W between the fully connected neurons in the first and second layers _2. Establish connection constraints for these two weights according to the following rules: mask ¹ , mask ² , mask ¹ , and mask ² are matrices filled with 1 or 0, where 1 and 0 respectively indicate whether the connection between neurons exists, according to The following rules determine the connection weight of i→j: a) If neuron i represents a single-sensory neuron, and neuron j represents a single-sensory neuron of the same modality, then the connection weight of i→j exists, b) If neuron i represents a single-sensory neuron, and neuron j represents a multi-sensory neuron, then the connection weight of i→j exists, c) If neuron i represents a multi-sensory neuron and neuron j represents a single-sensory neuron, then the connection weight of i→j exists, In other cases, the connection weight of i→j does not exist, 8. The method according to claim 7, characterized in that the complete multi-modal emotion recognition model in S6 includes each single-modal feature extraction network and a multi-modal recognition network; The multi-modal recognition network has a similar architecture to each single-modal recognition network, including two fully connected layers and one output layer. The difference is: a) Each fully connected layer neuron in the single-modal recognition network has a fixed number of neurons, but the number of neurons in the fully connected layer in the multi-modal recognition network is The change is the sum of the numbers of the two types of single-sensory neurons and multi-sensory neurons extracted. The calculation formula is: b) The layers in the multi-modal recognition network are sparsely connected, and the extracted connection constraints mask ¹ and mask ² are used, between the high-level features of each single modality and the neurons of the first fully connected layer The connection weight W ₁ and the connection weight W ₂ between the fully connected neurons in the first layer and the second layer, calculate the Hadamard product of mask ¹ and mask ² with W ₁ and W ₂ respectively, and calculate the Hadamard product between the layers. The connection is subject to regular restrictions, and the calculation formula is: Among them, * represents the Hadamard product, Represents the element of the i-th row and j-th column in the matrix of connection constraint mask ¹ , W _{1, i, j} represents the element of the i-th row and j-th column of the matrix of connection weight W ₁ , /> Represents the element in the i-th row and j-th column in the matrix of connection constraint mask ² , and W _{2, i, j} represents the element in the i-th row and j-th column in the matrix of connection weight W ₂ . 9. The method according to claim 1, characterized in that, in S6, a complete multi-modal emotion recognition model is trained, specifically including: The original multi-modal data samples are subjected to single-modal data preprocessing and primary feature extraction. The single-modal primary features are input into the single-modal feature extraction network to obtain single-modal high-level features. The single-modal high-level features are one-dimensionally After transformation, the spliced features are input to the multi-modal recognition network, and the entire model is trained using the DFA algorithm. During the training process, the weight of each single-modal feature extraction network is fixed, which is the weight after training in S3. Only the weights in the multimodal recognition network are trained. 10. A multi-modal emotion recognition device, characterized in that the device includes: The first acquisition module is used to preprocess the original multi-modal data sample and obtain the primary features of each single modality included in the original multi-modal data sample; The first extraction module is used to input the primary features of each single modality into the single-modality feature extraction network respectively, and extract the high-level features of each single modality; The first training module is used to convert the high-level features of each single modality into one dimension and then input them into each single-modality recognition network respectively. The pair consists of each single-modality feature extraction network and a single-modality recognition network. Single-modal model for training; The second acquisition module is used to obtain the weight of each single-modal feature extraction network from the trained single-modal model, and extract the activation degree of all neurons in each single-modal recognition network; The third acquisition module is used to determine the type of each neuron based on its activation degree, and obtain unique connection constraints between different neurons based on the type of each neuron; The second training module is used to establish a complete multi-modal emotion recognition model and train it based on the obtained weight of each single-modal feature extraction network and the extracted connection constraints, and obtain the trained multi-modal emotion recognition model. , for identifying multimodal emotions.