CN111616721A - Emotion recognition system and application based on deep learning and brain-computer interface - Google Patents

Abstract

The emotion recognition system comprises portable electroencephalogram collection equipment, an emotion electroencephalogram classification module and an emotion classification display module which are sequentially connected, wherein the portable electroencephalogram collection equipment collects emotion EEG electroencephalograms from the brains of a testee, the emotion electroencephalogram classification module analyzes the emotion EEG electroencephalograms, a relation model between the emotion EEG electroencephalograms and emotions is established, emotion classification is carried out on the input emotion EEG electroencephalograms through the relation model, and the emotion classification display module displays the recognized emotion types and prompts the emotion state of the testee. The emotion recognition system based on deep learning and brain-computer interface and the application thereof can realize accurate acquisition, effective identification and correct classification of emotion electroencephalogram signals, visually prompt emotion states and realize the function of emotion state monitoring. The invention directly establishes the corresponding relation between the electroencephalogram signals and the emotion types and realizes the feedback of the emotion.

Description

Emotion recognition system and application based on deep learning and brain-computer interface

technical field

The present invention relates to an emotion recognition system. In particular, it relates to an emotion recognition system and application based on deep learning and brain-computer interface.

Background technique

Emotion is the reaction of people when they are stimulated by external things, and it is a broad and complex physiological and psychological state. There are many kinds of human emotional states, which are roughly defined by psychologist Ekman as happiness, anger, surprise, fear, disgust and sadness. In fact, there are many other emotional states such as shame, arrogance, disappointment, anxiety, etc. A person's emotional state often reflects a person's attitude and views on external things. Emotion recognition can help people improve the safety of device use and analyze potential emotional factors in daily behavior. For example, in rehabilitation medicine, it can help doctors diagnose and prevent problems such as depression and post-traumatic stress disorder. At the same time, according to the patient's emotional response, it is convenient for medical staff to provide better medical care to assist the patient's recovery. In terms of transportation, emotional detection can be used to analyze the mental state of the driver, such as whether it is fatigued, how awake, whether it is nervous and anxious, etc., to ensure driving safety. In the education industry, students' strengths can be developed through students' emotional responses to different subjects, and more targeted tutoring can be carried out. At the same time, when the students' brains are under heavy load and show fatigue, teachers are prompted to activate the classroom or take a short break to ensure classroom efficiency. Accurate recognition of emotions can improve our quality of life in many ways.

Human emotions can be reflected in various ways, such as facial expressions, voice intonation, eye movements and so on. At this stage, emotion recognition can use modalities including expressions, voices, actions, and physiological signals. Among them, physiological signals, especially electroencephalogram (EEG) signals are not easily affected by human factors. Due to the stability of EEG signals, EEG-based emotion classification has received extensive attention. However, there are still many challenges in using EEG signals for emotion analysis in practical applications. First of all, the EEG acquisition process is relatively difficult. To obtain stable EEG signals, there are certain requirements for acquisition equipment, subjects, and the external environment. Although implantable and semi-implantable EEG electrodes can obtain more stable signals, their operation is complicated and the feasibility of daily use is low. At present, non-implantable EEG electrodes are widely used in the industry. Although they have the advantages of portability and simple operation, they are easily affected by external noise. Secondly, the EEG signal has the characteristics of weak intensity and strong background noise. To obtain a real and effective EEG signal requires advanced preprocessing technology. At the same time, in the classification work, some processing techniques are needed to extract features from a large number of signal samples to establish the correspondence between signals and emotional states, and this process requires high accuracy of the processing methods.

Recently, complex network theory, as a multidisciplinary fusion method, provides a new approach for complex system analysis. Among them, multi-layer networks have advantages in analyzing spatiotemporal datasets due to their multi-scale characteristics. The human brain is a complex system, and complex networks are undoubtedly a powerful tool for analyzing the interaction patterns of human brain signals in multiple spaces and scales. Among them, the method of using EEG signals to establish a brain network for analysis has also attracted more and more attention. Brain electrodes are set as nodes, and the connections between nodes are determined through various correlation measures to establish a brain network. By studying the characteristics of brain networks, we can deepen the understanding of the characteristics of EEG signals.

With the improvement of computer processing speed and computing power, it is possible to use deep learning to process large data samples. As a cutting-edge technology in the field of artificial intelligence, deep learning is an end-to-end learning method that can automatically learn and extract abstract features of input samples without manual operation analysis, which greatly improves processing efficiency. EEG signal data has the characteristics of large quantity, many features, and rapid changes. Using deep learning methods to analyze EEG data has obvious advantages. Among them, the Convolutional Neural Network (CNN) has the characteristics of weight sharing and local perception, which effectively reduces the number and complexity of model parameters, making the model easier to train. At the same time, it is closer to the actual biological neural network, has invariance to the degree of variation in the form of translation, rotation, scaling, etc., and can effectively learn the corresponding features from a large number of samples. As a deep learning model based on convolutional neural network, Densenet has the characteristics of parameter saving, high computational efficiency, and certain resistance to overfitting. It has great potential in analyzing large data samples.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide an emotion recognition system and application based on deep learning and brain-computer interface, which can achieve accurate acquisition, effective identification and correct classification of emotional EEG signals.

The technical scheme adopted in the present invention is: an emotion recognition system based on deep learning and brain-computer interface, comprising a portable EEG acquisition device, an emotion EEG classification module and an emotion classification display module connected in sequence, the portable EEG collection The device collects emotional EEG signals from the subject's brain, and the emotional EEG classification module analyzes the emotional EEG signals to establish a relationship model between the emotional EEG signals and emotions, and through the relationship model, the input The emotional EEG electroencephalogram signal is used for emotional classification, and the emotional classification display module displays the recognized emotional types, and prompts the emotional state of the subjects.

An application of an emotion recognition system based on deep learning and brain-computer interface, comprising the following steps:

1) The user watches different emotional videos and generates three different emotions under the stimulation of the video, namely positive, neutral and negative;

2) Use the portable EEG acquisition device to collect the emotional EEG signals of the user under three different emotions, and train the emotion classification deep learning model according to the emotional EEG EEG signals and the corresponding emotional labels;

3) Use a portable EEG acquisition device to collect the new emotional EEG signal of the user to be identified, and input it into a dual-input deep learning model for emotion classification and detection, and perform emotion classification on the new emotional EEG signal;

4) Display the current emotion type through the display, and prompt the user's emotional state.

The emotion recognition system and application based on deep learning and brain-computer interface of the present invention can realize the accurate acquisition, effective identification and correct classification of emotional EEG signals, intuitively prompt emotional state, and realize the function of emotional state monitoring. The present invention directly establishes the corresponding relationship between the EEG signal and the emotion type, so as to realize the feedback of emotion.

Description of drawings

Fig. 1 is the composition block diagram of the emotion recognition system based on deep learning and brain-computer interface of the present invention;

Fig. 2 is a block diagram of the structure of the portable EEG acquisition device in the present invention;

Fig. 3 is the flow chart of constructing multi-layer brain complex network in the present invention;

4 is a schematic structural diagram of a dual-input deep learning model in the present invention;

Fig. 5 is the application schematic diagram of the emotion recognition system based on deep learning and brain-computer interface of the present invention;

Detailed ways

The emotion recognition system and application based on deep learning and brain-computer interface of the present invention will be described in detail below with reference to the embodiments and accompanying drawings.

As shown in Figure 1, the emotion recognition system based on deep learning and brain-computer interface of the present invention includes a portable EEG acquisition device 1, an emotion EEG classification module 2 and an emotion classification display module 3 connected in sequence. The acquisition device 1 collects emotional EEG signals from the subject's brain, and the emotional EEG classification module 2 analyzes the emotional EEG signals, and establishes a relationship model between the emotional EEG signals and emotions. , and perform emotion classification on the input emotional EEG signal, and the emotion classification display module 3 displays the recognized emotion types, and prompts the subject's emotional state.

As shown in FIG. 2 , the portable EEG acquisition device 1 of the present invention includes: a brain electrode cap and its adapter cable 11 connected in sequence for collecting emotional EEG signals, amplifying and The converted bioelectric signal acquisition module 12, the FPGA processor 13 for controlling the acquisition of emotional EEG signals and transmitting the emotional EEG signals to the emotional EEG classification module 2 through the USB communication circuit 14, and respectively connecting the bioelectric signals The system power supply circuit 15 of the acquisition module 12 and the FPGA processor 13, wherein,

The brain electrode cap and the brain electrode cap in the patch cord 11 collect emotional EEG signals of different brain regions, and are connected with the bioelectric signal acquisition module 12 through the patch cord and the DSUB37 interface for the acquisition of bioelectric signals. and transmission;

The bioelectric signal acquisition module 12 is composed of several pieces of analog input modules with a high common-mode rejection ratio for receiving the voltage signals collected by the EEG cap, and a low-noise programmable gain amplifier PGA for amplifying the brain voltage signals. It is composed of a bioelectric signal acquisition chip of a high-resolution synchronous sampling analog-to-digital converter ADC that converts analog signals into digital signals;

The FPGA processor 13 is used to adjust the acquisition mode and parameters of the bioelectric signal acquisition module 12 and to control the USB communication circuit 14 to transmit the emotional EEG signal data to the emotional EEG classification module 2;

The output of the USB communication circuit 14 is connected to the input end of the emotional EEG classification module 2. The USB communication circuit 14 works in an asynchronous FIFO mode, with a maximum transmission rate of 8MB/s, and periodically under the control of the FPGA processor 13. Send the collected emotional EEG signals to the emotional EEG classification module 2 in the form of data packets;

The system power supply circuit 15 has an input voltage of 5V, is powered by a USB interface, and provides the operating voltages of different chips of the system through a voltage conversion module.

The portable EEG acquisition device 1 collects the subject's emotional EEG signal, which is to collect the subject's FP1, FPZ, FP2, AF3, AF4, F7, F5, F3, F1, FZ, F2,F4,F6,F8,FT7,FC5,FC3,FC1,FCZ,FC2,FC4,FC6,FT8,T7,C5,C3,C1,CZ,C2,C4,C6,T8,TP7,CP5,CP3, CP1,CPZ,CP2,CP4,CP6,TP8,P7,P5,P3,P1,PZ,P2,P4,P6,P8,PO7,PO5,PO3,POZ,PO4,PO6,PO8,CB1,O1,OZ, The emotional EEG signals of 62 electrodes in O2 and CB2; the electrode distribution of the brain electrode cap conforms to the 10/20 international standard lead; specifically, video clips are used as emotional stimulation sources, and the subjects are collected to generate corresponding emotional states after watching the video The emotional EEG EEG signals; the corresponding emotional states are positive emotional state, negative emotional state and neutral emotional state, and the collection process is:

(1) The subjects watched video clips corresponding to three different emotions, and generated emotional EEG signals during the stimulation of the video source;

(2) Each time a video is watched, the portable EEG acquisition device completes the collection of emotional EEG signals, and sets labels, tags and emotional EEG signals for each collected emotional EEG signal according to the video used for stimulation The electrical signals together constitute the sample set.

The operation of the emotional EEG classification module 2 of the present invention specifically includes the following steps:

c＝1,2,…,62构成样本集，其中c代表电极序号，L为信号采集的长度，X_c,g表示第c个电极采样到的第g个数据点的数值；1) Use a portable EEG acquisition device to obtain emotional EEG signals of three emotional states

c=1,2,...,62 constitute a sample set, where c represents the electrode serial number, L is the length of signal acquisition, X _c,g represents the value of the gth data point sampled by the cth electrode;

进行滑动窗口划分，得到情绪EEG脑电信号片段，对情绪EEG脑电信号片段进行连续小波变换，将原情绪EEG脑电信号片段分为五个频段，在五个频段δ,θ,α,β,γ内分别构建脑复杂网络A^f,f＝δ,θ,α,β,γ，最终得到多层脑复杂网络；2) EEG EEG signals for emotions

Divide the sliding window to obtain emotional EEG signal fragments, perform continuous wavelet transform on the emotional EEG EEG signal fragments, and divide the original emotional EEG EEG signal fragments into five frequency bands. , γ to build a complex brain network A ^f , f = δ, θ, α, β, γ, and finally obtain a multi-layer brain complex network;

The five frequency bands are: δ (1-3Hz), θ (4-7Hz), α (8-12Hz), β (13-30Hz), and γ (31-50Hz) five frequency bands, multi-layer The construction of the complex brain network is shown in Figure 3, which includes:

通过长度为l的无重叠滑动窗口进行数据分割，滑动窗口的滑动步长为b，得到一系列滑动窗口数据，其中第j个滑动窗口数据表示为

表示第c个频段中第j个滑动窗口中的第g₁个数据点；(2.1) For the collected emotional EEG signals

The data is divided by a non-overlapping sliding window of length l, and the sliding step size of the sliding window is b, and a series of sliding window data are obtained, in which the jth sliding window data is expressed as

represents the g _- th data point in the j-th sliding window in the c-th frequency band;

f＝δ,θ,α,β,γ；(2.2) For the data segment obtained by each sliding window, use continuous wavelet transform to divide each emotional EEG signal segment into 5 frequency bands, and obtain the emotional EEG EEG signal after frequency division

f = δ, θ, α, β, γ;

将每个通道设置为一个节点，节点之间的连边通过皮尔逊相关系数(Pearson correlation coefficient，PCC)来推断，以此来建立脑复杂网络，包括：(2.3) EEG EEG signals in a frequency band

Each channel is set as a node, and the edges between nodes are inferred by the Pearson correlation coefficient (PCC) to build a complex brain network, including:

(2.3.1) Calculation of the Pearson correlation coefficient between channels, the Pearson correlation coefficient can measure the similarity between two variable sequences, and the value ranges from -1 to 1. The Pearson correlation coefficient between the two channels ρ _{X, Y} is calculated as follows:

Among them, σ _X and σ _Y are the standard deviation of the emotional EEG signals of channel X and channel Y, respectively, and the Pearson correlation coefficient between the two channels is calculated to obtain the adjacency matrix between channels;

(2.3.2) Sort the weights of the Pearson correlation coefficient values between channels, remove 50% of the edges with smaller weights, and obtain the complex brain network in this frequency band;

(2.4) Step (2.3) is repeated in the five frequency bands to obtain the complex brain network A ^δ , A ^θ , A ^α , A ^β , A ^γ corresponding to the five frequency bands δ, θ, α, β, γ, and then form Multilayered brain complex network.

3) Select the 30 nodes with the largest differences between different types of emotional brain complex networks as key nodes, and reconstruct the multi-layer brain complex network according to the key nodes; including:

其中k_i表示与节点i相邻节点的数量，e(i)表示节点i实际连边数量；(3.1) Calculate the aggregation coefficient of each network node

where k _i represents the number of nodes adjacent to node i, and e(i) represents the actual number of edges connected to node i;

q1≠q2，其中

表示不同类别情绪脑复杂网络的节点i的聚集系数，q1和q2代表两种不同的情绪状态，对M值进行从大到小排序，选取M值较大的30个节点作为关键节点，通过保留脑复杂网络中关键节点与关键节点之间的连边，并去除其他连边，来重构多层脑复杂网络。(3.2) Statistical difference in clustering coefficient of node i among three different emotions

q1≠q2, where

Represents the clustering coefficient of node i of different types of emotional brain complex networks, q1 and q2 represent two different emotional states, sort M values from large to small, select 30 nodes with larger M values as key nodes, and keep The connection between the key nodes and the key nodes in the complex brain network is removed, and other connections are removed to reconstruct the multi-layer complex brain network.

4) Build a convolutional neural network branch of a dual-input deep learning model. The input of the convolutional neural network branch is a multi-layer brain complex network and corresponding labels, and the convolutional neural network branch is composed of five layers of the same convolutional neural network. Model composition, each layer of convolutional neural network model corresponds to one of the five frequency bands δ, θ, α, β, γ of the multi-layer brain complex network; each layer of the convolutional neural network model includes sequentially connected :

(4.1) data input layer, the input is the corresponding frequency band brain complex network A ^f ;

(4.2) The first convolution layer, the size of the convolution kernel is 5×5, the number of convolution kernels is 8, and the activation function is the ReLU function;

(4.3) The first average pooling layer, the size of the pooling kernel is 2×2;

(4.4) The second convolution layer, the size of the convolution kernel is 5×5, the number of convolution kernels is 16, and the activation function is ReLU;

(4.5) The second average pooling layer, the size of the pooling kernel is 2×2;

(4.6) Dropout layer, randomly select the neurons of the previous layer with probability p, so that the neurons do not output, and reduce the over-fitting phenomenon, for δ, θ, α frequency bands p=0.03, for β, γ frequency bands p=0.01 ;

(4.7) Batch normalization layer;

(4.8) Flatten layer, which makes the multi-dimensional input one-dimensional;

(4.9) The fully connected layer, the activation function is ReLU, the L1 norm is selected as the regularization term, and the L1 norm is set to 0.0005.

5) Build a dense convolutional network (DenseNet) branch of the dual-input deep learning model. The input of the dense convolutional network branch is the emotional EEG signal and the corresponding label after 0-50Hz filtering using wavelet transform;

The dense convolutional network branch of the dual-input deep learning model includes sequentially connected:

(5.1) Data input layer, the input data is the emotional EEG signal of 0-50Hz;

(5.2) Convolution layer, the size of the convolution kernel is 1×62, and the number of convolution kernels is 24;

(5.3) The first dense block is composed of three convolution blocks connected in sequence, each convolution block includes a batch normalization layer and a convolution layer, and the convolution kernel size of each convolution layer is 3× 1. The number of convolution kernels of the three convolutional layers is 36, 48, and 60 respectively;

(5.4) The first transition block consists of a convolution layer with a convolution kernel size of 1×1 and a convolution kernel number of 72, and a convolution layer with a convolution kernel size of 2×1 and the number of convolution kernels 72 , and a 2×2 pooling layer;

(5.5) The second dense block is composed of three convolution blocks connected in sequence, each convolution block includes a batch normalization layer and a convolution layer, and the convolution kernel size of each convolution layer is 3× 1. The number of convolution kernels of the three convolutional layers is 72, 84, and 96 respectively;

(5.6) The second transition block consists of a convolution layer with a convolution kernel size of 1×1 and a convolution kernel number of 108, a convolution layer with a convolution kernel size of 2×1 and the number of convolution kernels 108 , and a 2×2 pooling layer;

(5.7) The third dense block is composed of three convolution blocks connected in sequence. Each convolution block includes a batch normalization layer and a convolution layer, and the size of the convolution kernel of each convolution layer is 3× 1. The number of convolution kernels of the three convolutional layers is 108, 120, and 132 respectively;

(5.8) Global pooling layer with a pooling kernel size of 50×1.

其中h＝1…H，e为自然对数值，z_h为第h个神经元的输出，式中分母充当正则项作用，使得

使用Keras框架实现深度学习模型的搭建，得到双输入深度学习模型；6) Connect the output of the convolutional neural network branch and the output of the dense convolutional network branch through a series layer, the series layer is connected to another fully connected layer, the fully connected layer outputs the classification result, and uses H neural networks. Meta representation, the classification layer selects Softmax as the activation function, the Softmax function is essentially a normalized exponential function, defined as

where h=1...H, e is the natural logarithm value, z _h is the output of the hth neuron, and the denominator acts as a regular term, so that

Use the Keras framework to build a deep learning model and obtain a dual-input deep learning model;

7) Divide the collected emotional EEG signals in three emotional states into a training set and a validation set, use the training set to train the dual-input deep learning model, and use the validation set to perform the training on the dual-input deep learning model. After verification, a dual-input deep learning model for emotion classification and detection is finally obtained. include:

For the training of the dual-input deep learning model, the Adam optimization algorithm is used. By minimizing the cross-entropy loss function, various weights and deviations in the dual-input deep learning model are optimized, and the optimal value when the loss function of the validation set is minimized is recorded. For the dual-input deep learning model, the learning rate of the optimal dual-input deep learning model is set to 0.001, a total of 100 cycles of cyclic training are performed, and the batch size is 100. Finally, a dual-input deep learning model for emotion classification and detection is obtained.

In the example of the present invention, the EEG data samples during the training of the dual-input deep learning model came from a total of 15 experimenters, with an average age of 23 years, including 7 women. The EEG acquisition frequency was 200 Hz. 50 minutes of data were collected during the rest time. Considering individual differences among experimenters, a two-input deep learning model was trained with the EEG signals of each experimenter. During dual-input deep learning model training, 80% of the sample set is used as the training model parameters, 10% is used as the validation set to get the best model during training, and 10% of the sample set is used as the test set to determine Model final performance. In the performance test, the model of 15 experimenters achieved an average classification accuracy of 97.27% in the emotion three-classification task.

The emotion recognition system based on deep learning and brain-computer interface of the present invention does not require new adjustment for the dual-input deep learning model trained by the EEG sample data of each user. In the emotion classification detection task, as long as the user's new EEG sample data is input, the dual-input deep learning model can determine the corresponding emotion type.

As shown in Figure 5, the application of the emotion recognition system based on deep learning and brain-computer interface of the present invention includes the following steps:

1) The user watches different emotional videos and generates three different emotions under the stimulation of the video, namely positive, neutral and negative;

2) Use the portable EEG acquisition device to collect the emotional EEG signals of the user under three different emotions, and train the emotion classification deep learning model according to the emotional EEG EEG signals and the corresponding emotional labels;

3) Use a portable EEG acquisition device to collect the new emotional EEG signal of the user to be identified, and input it into a dual-input deep learning model for emotion classification and detection, and perform emotion classification on the new emotional EEG signal;

4) Display the current emotion type through the display, and prompt the user's emotional state.

Claims (10)

1. an emotion recognition system based on deep learning and a brain-computer interface, comprising a portable EEG acquisition device (1), an emotion EEG classification module (2) and an emotion classification display module (3) connected successively, it is characterized in that, The portable EEG acquisition device (1) collects emotional EEG signals from the subject's brain, and the emotional EEG classification module (2) analyzes the emotional EEG signals to establish emotional EEG signals and emotional EEG signals. The relationship model is used to classify the emotions of the input emotional EEG signals through the relationship model, and the emotion classification display module (3) displays the identified emotion types and prompts the subject's emotional state. 2. The emotion recognition system based on deep learning and brain-computer interface according to claim 1, wherein the portable EEG acquisition device (1) comprises: sequentially connected for acquiring emotional EEG EEG signals The brain electrode cap and its adapter cable (11), a bioelectric signal acquisition module (12) for amplifying and converting emotional EEG signals, for controlling the acquisition of emotional EEG signals and communicating via a USB communication circuit (14) An FPGA processor (13) for transmitting emotional EEG signals to the emotional EEG classification module (2), and a system power supply circuit (15) respectively connected to the bioelectrical signal acquisition module (12) and the FPGA processor (13), wherein , The brain electrode cap and the brain electrode cap in the adapter cable (11) collect emotional EEG signals of different brain regions, and are connected with the bioelectric signal acquisition module (12) through the adapter cable and the DSUB37 interface, and are used for biological Collection and transmission of electrical signals; The bioelectric signal acquisition module 12 is composed of several pieces of analog input modules with high common-mode rejection ratio for receiving the voltage signals collected by the EEG cap, and low-noise programmable gain amplifiers ( PGA) and a bioelectrical signal acquisition chip for converting analog signals into digital signals with high-resolution simultaneous sampling analog-to-digital converters (ADC); The FPGA processor 13 is used to adjust the acquisition mode and parameters of the bioelectric signal acquisition module (12) and to control the USB communication circuit (14) to transmit the emotional EEG signal data to the emotional EEG classification module (2); The output of the USB communication circuit (14) is connected to the input end of the emotional EEG classification module (2). The USB communication circuit (14) works in an asynchronous FIFO mode with a maximum transmission rate of 8MB/sec. 13) periodically sending the collected emotional EEG signals to the emotional EEG classification module (2) in the form of data packets; The system power supply circuit (15) has an input voltage of 5V, is powered by a USB interface, and provides working voltages of different chips of the system through a voltage conversion module. 3. The emotion recognition system based on deep learning and brain-computer interface according to claim 1, is characterized in that, described portable EEG acquisition device (1) collects the subject's emotional EEG EEG signal, which is to collect the subject's emotional EEG signal. which correspond to FP1, FPZ, FP2, AF3, AF4, F7, F5, F3, F1, FZ, F2, F4, F6, F8, FT7, FC5, FC3, FC1, FCZ, FC2, FC4, FC6 of the brain electrode cap ,FT8,T7,C5,C3,C1,CZ,C2,C4,C6,T8,TP7,CP5,CP3,CP1,CPZ,CP2,CP4,CP6,TP8,P7,P5,P3,P1,PZ,P2 ,P4,P6,P8,PO7,PO5,PO3,POZ,PO4,PO6,PO8,CB1,O1,OZ,O2,CB2, a total of 62 emotional EEG signals; the electrode distribution of the brain electrode cap conforms to 10/ 20 international standard leads; specifically, video clips are used as emotional stimulation sources to collect emotional EEG signals that generate corresponding emotional states after the subjects watch the videos; the corresponding emotional states are positive emotional state, negative emotional state and moderate emotional state. Sexual emotional state, the collection process is: (1) The subjects watched video clips corresponding to three different emotions, and generated emotional EEG signals during the stimulation of the video source; (2) Each time a video is watched, the portable EEG acquisition device completes the collection of emotional EEG signals, and sets labels, tags and emotional EEG signals for each collected emotional EEG signal according to the video used for stimulation The electrical signals together constitute the sample set. 4. the emotion recognition system based on deep learning and brain-computer interface according to claim 1, is characterized in that, the operation of described emotion EEG classification module (2) specifically comprises the steps: 1) Use a portable EEG acquisition device to obtain emotional EEG signals of three emotional states

Constitute a sample set, where c represents the electrode serial number, L is the length of signal acquisition, X _{c, g} represent the value of the g-th data point sampled by the c-th electrode; 2) EEG EEG signals for emotions

Divide the sliding window to obtain emotional EEG signal fragments, perform continuous wavelet transform on the emotional EEG signal fragments, and divide the original emotional EEG EEG signal fragments into five frequency bands, in the five frequency bands δ, θ, α, β , γ to build a complex brain network A ^f , f = δ, θ, α, β, γ, and finally obtain a multi-layer brain complex network; 3) Select the 30 nodes with the largest differences between different types of emotional brain complex networks as key nodes, and reconstruct the multi-layer brain complex network according to the key nodes; 4) Build a convolutional neural network branch of a dual-input deep learning model. The input of the convolutional neural network branch is a multi-layer brain complex network and corresponding labels, and the convolutional neural network branch is composed of five layers of the same convolutional neural network. Model composition, each layer of convolutional neural network model corresponds to one of the five frequency bands δ, θ, α, β, γ of the multi-layer brain complex network; 5) Build a dense convolutional network branch of a dual-input deep learning model. The input of the dense convolutional network branch is the emotional EEG signal and corresponding label after 0-50Hz filtering using wavelet transform; 6) Connect the output of the convolutional neural network branch and the output of the dense convolutional network branch through a series layer, the series layer is connected to another fully connected layer, the fully connected layer outputs the classification result, and uses H neural networks. Meta representation, the classification layer selects Softmax as the activation function, the Softmax function is essentially a normalized exponential function, defined as

where h=1...H, e is the natural logarithm value, z _h is the output of the hth neuron, and the denominator acts as a regular term, so that

Use the Keras framework to build a deep learning model and obtain a dual-input deep learning model; 7) Divide the collected emotional EEG signals in three emotional states into a training set and a validation set, use the training set to train the dual-input deep learning model, and use the validation set to perform the training on the dual-input deep learning model. After verification, a dual-input deep learning model for emotion classification and detection is finally obtained. 5. The emotion recognition system based on deep learning and brain-computer interface according to claim 4, wherein the five frequency bands described in step 2) are respectively: δ (1-3Hz), θ (4-7Hz) ), α (8-12Hz), β (13-30Hz), and γ (31-50Hz) five frequency bands, the construction of multi-layer brain complex network specifically includes: (2.1) For the collected emotional EEG signals

The data is divided by a non-overlapping sliding window of length l, and the sliding step size of the sliding window is b, and a series of sliding window data are obtained, in which the jth sliding window data is expressed as

represents the g _- th data point in the j-th sliding window in the c-th frequency band; (2.2) For the data segment obtained by each sliding window, use continuous wavelet transform to divide each emotional EEG signal segment into 5 frequency bands, and obtain the emotional EEG EEG signal after frequency division

(2.3) EEG EEG signals in a frequency band

Each channel is set as a node, and the connections between nodes are inferred by the Pearson correlation coefficient to build a complex brain network, including: (2.2.1) Calculation of the Pearson correlation coefficient between channels, the Pearson correlation coefficient can measure the similarity between two variable sequences, the value range is -1 to 1, the Pearson correlation coefficient ρ _{X between the two channels, Y} is calculated as follows:

Among them, σ _X and σ _Y are the standard deviation of the emotional EEG signals of channel X and channel Y, respectively, and the Pearson correlation coefficient between the two channels is calculated to obtain the adjacency matrix between channels; (2.2.2) Rank the values of the Pearson correlation coefficient between channels by weight, remove 50% of the edges with smaller weights, and obtain the complex brain network in this frequency band; (2.4) Step (2.3) is repeated in the five frequency bands to obtain the complex brain network A ^δ , A ^θ , A ^α , A ^β , A ^γ corresponding to the five frequency bands δ, θ, α, β, γ, and then form Multilayered brain complex network. 6. The emotion recognition system based on deep learning and brain-computer interface according to claim 4, is characterized in that, step 3) comprises: (3.1) Calculate the aggregation coefficient of each network node

where k _i represents the number of nodes adjacent to node i, and e(i) represents the actual number of edges connected to node i; (3.2) Statistical difference in clustering coefficient of node i among three different emotions

q1≠q2, where Ciq1 and Ciq2 represent the aggregation coefficients of nodes i of different types of emotional brain complex networks, q1 and q2 represent two different emotional states, sort the M values from large to small, and select 30 with larger M values As the key node, only the connections between the key nodes and the key nodes in the complex brain network are retained, and other connections are removed to reconstruct the multi-layer complex brain network. 7. The emotion recognition system based on deep learning and brain-computer interface according to claim 4, is characterized in that, each layer of convolutional neural network model described in step 4) comprises sequentially connected: (4.1) data input layer, the input is the corresponding frequency band brain complex network A ^f ; (4.2) The first convolution layer, the size of the convolution kernel is 5×5, the number of convolution kernels is 8, and the activation function is the ReLU function; (4.3) The first average pooling layer, the size of the pooling kernel is 2×2; (4.4) The second convolution layer, the size of the convolution kernel is 5×5, the number of convolution kernels is 16, and the activation function is ReLU; (4.5) The second average pooling layer, the size of the pooling kernel is 2×2; (4.6) Dropout layer, randomly select the neurons of the previous layer with probability p, so that the neurons do not output, reduce the over-fitting phenomenon, for δ, θ, α frequency bands p=0.03, for β, γ frequency bands p=0.01 ; (4.7) Batch normalization layer; (4.8) Flatten layer, which makes the multi-dimensional input one-dimensional; (4.9) The fully connected layer, the activation function is ReLU, the L1 norm is selected as the regularization term, and the L1 norm is set to 0.0005. 8. The emotion recognition system based on deep learning and brain-computer interface according to claim 4, is characterized in that, the dense convolutional network branch of the double-input deep learning model described in step 5) comprises sequentially connected: (5.1) Data input layer, the input data is the emotional EEG signal of 0-50Hz; (5.2) Convolution layer, the size of the convolution kernel is 1×62, and the number of convolution kernels is 24; (5.3) The first dense block is composed of three convolution blocks connected in sequence, each convolution block includes a batch normalization layer and a convolution layer, and the convolution kernel size of each convolution layer is 3× 1. The number of convolution kernels of the three convolutional layers is 36, 48, and 60 respectively; (5.4) The first transition block consists of a convolution layer with a convolution kernel size of 1×1 and a convolution kernel number of 72, and a convolution layer with a convolution kernel size of 2×1 and the number of convolution kernels 72 , and a 2×2 pooling layer; (5.5) The second dense block is composed of three convolution blocks connected in sequence, each convolution block includes a batch normalization layer and a convolution layer, and the convolution kernel size of each convolution layer is 3× 1. The number of convolution kernels of the three convolutional layers is 72, 84, and 96 respectively; (5.6) The second transition block consists of a convolution layer with a convolution kernel size of 1×1 and a convolution kernel number of 108, a convolution layer with a convolution kernel size of 2×1 and the number of convolution kernels 108 , and a 2×2 pooling layer; (5.7) The third dense block is composed of three convolution blocks connected in sequence. Each convolution block includes a batch normalization layer and a convolution layer, and the size of the convolution kernel of each convolution layer is 3× 1. The number of convolution kernels of the three convolutional layers is 108, 120, and 132 respectively; (5.8) Global pooling layer with a pooling kernel size of 50×1. 9. the emotion recognition system based on deep learning and brain-computer interface according to claim 4, is characterized in that, in step 7), to the training of dual-input deep learning model, select Adam optimization algorithm, by minimizing cross entropy loss function , optimize the various weights and deviations in the dual-input deep learning model, record the optimal dual-input deep learning model when the loss function of the validation set is the smallest, and set the learning rate of the optimal dual-input deep learning model to 0.001. After 100 cycles of cyclic training, the Batchsize is 100, and finally a dual-input deep learning model for emotion classification and detection is obtained. 10. The application of the emotion recognition system based on deep learning and brain-computer interface according to claim 1, is characterized in that, comprises the steps: 1) The user watches different emotional videos and generates three different emotions under the stimulation of the video, namely positive, neutral and negative; 2) Use the portable EEG acquisition device to collect the emotional EEG signals of the user under three different emotions, and train the emotion classification deep learning model according to the emotional EEG EEG signals and the corresponding emotional labels; 3) Use a portable EEG acquisition device to collect the new emotional EEG signal of the user to be identified, and input it into a dual-input deep learning model for emotion classification and detection, and perform emotion classification on the new emotional EEG signal; 4) Display the current emotion type through the display, and prompt the user's emotional state.

Images