CN113378650B - An Emotion Recognition Approach Based on Brain Power Imaging and Regularized Common Spatial Patterns - Google Patents

Abstract

The invention relates to the field of signal feature extraction, in particular to an emotion recognition method based on an electroencephalogram source imaging and regularization common space mode, which comprises the steps of collecting an electroencephalogram signal and preprocessing the electroencephalogram signal; reconstructing cerebral cortex nerve electrical activity by constructing an electroencephalogram source image; projecting the reconstructed EEG time series onto a Brodman partition by a minimum modulus algorithm, wherein the Brodman partition comprises 26 spatial interest areas, and a brain function connection matrix is constructed by utilizing the mutual information of the 26 spatial interest areas; constructing a covariance matrix of a generalized sample based on the brain function connection matrix in a regularization mode, and extracting a characteristic vector; inputting the characteristics of the historical data into a classifier for training, and inputting the characteristic vector of the data to be classified into the classifier to obtain electroencephalogram emotion classification; the method reduces estimation deviation, improves estimation stability, and improves the accuracy of the finally classified result.

Description

An Emotion Recognition Approach Based on Brain Power Imaging and Regularized Common Spatial Patterns

technical field

The invention relates to the field of signal feature extraction, in particular to an emotion recognition method based on brain power imaging and a regularized co-spatial pattern.

Background technique

Sentiment analysis is an important part of human-computer interaction. Electroencephalography (EEG) signals reflect cortical neural activity under cognitive tasks, and are increasingly valued in emotion recognition due to their high temporal resolution and non-invasiveness.

At present, the commonly used EEG-based emotion recognition methods first extract the attribute features of the EEG signal in the time domain and frequency domain, and then use the pattern classifier to perform emotion recognition. However, the lower spatial resolution of scalp EEG signals due to volumetric effects also limits the performance of EEG emotion recognition based on scalp EEG signals.

In addition, in practical applications, emotion recognition across subjects is a very important research content. Previously, Chinese patent CN110070105B published an EEG emotion recognition method based on meta-learning instances for rapid screening. This patent splices the feature vectors of EEG data of each electrode to obtain the feature vector to be recognized, and then uses the trained emotion recognition model. According to the feature vector to be identified, the corresponding emotion label is obtained to realize the emotion recognition across subjects. There are still some practical problems with this method. For example, the EEG data of different subjects is different, which makes it difficult to obtain a general model that can be used across subjects: given the non-stationary nature of EEG, the EEG distribution of the same subject changes over time, making it difficult to obtain a model that can be used for a long time. Since human brain cognitive behavior is the result of the synergy of several brain regions and is related to the interaction of brain regions, the brain network contains rich spatiotemporal classification information. This method ignores the classification information contained in the brain network, and does not effectively extract the classification features of the cortical brain network. At the same time, in EEG-based cross-subject emotion recognition research, how to make full use of the information of the existing subjects' EEG data is still an urgent problem to be solved.

SUMMARY OF THE INVENTION

In order to improve the accuracy of EEG signal classification, the present invention proposes an emotion recognition method based on brain power imaging and regularized co-spatial pattern, including the following steps:

Collect EEG signals and preprocess the EEG signals;

Use the Bayesian least mode algorithm to process and process the EEG signals, and reconstruct the neural electrical activity of the cerebral cortex by building brain power imaging;

Through the least-modulus algorithm, the reconstructed EEG time series is projected onto the Brodman partition, which includes 26 regions of spatial interest. After flipping the source signals in opposite directions, all source signals in the 26 regions of spatial interest are The time series is averaged;

Using the mutual information of 26 spatial regions of interest to construct a brain functional connectivity matrix;

Based on the brain functional connectivity matrix, the covariance matrix of the generalized samples is constructed by regularization, and the optimal spatial filter is obtained to maximize the variance value difference between the two types of signals, so as to obtain the eigenvectors with higher discrimination;

The features of the historical data are input into the classifier for training, and the feature vector of the data to be classified is input into the classifier to obtain the EEG emotion classification.

Further, reconstructing the neural electrical activity of the cerebral cortex specifically includes the following steps:

According to the propagation law of the electromagnetic field in the biological conductor, the linear relationship expression between the EEG potential distribution on the scalp surface and the endogenous spatial signal of the human brain was constructed;

Perform spatial whitening on the obtained linear relationship expression;

According to a prior distribution of the given cerebral cortex source signal and the Bayesian formula, calculate the posterior distribution of the source signal;

The maximum a posteriori estimate of the source signal is estimated by using the minimum modulus solution, that is, the source structure with the smallest energy is selected as the final source signal estimation by the minimum modulus solution, and the source signal estimation is used as the source signal for human brain imaging.

Further, using the minimum modulus solution to estimate the maximum a posteriori estimate of the source signal, the maximum a posteriori estimate of the source signal is expressed as:

为源信号的最大后验估计；p(S|B)为源信号S的后验分布；p(S)为源信号先验分布；L为导联矩阵；B为大脑头皮表面的脑电信号数据；λ为正则参数，I为单位矩阵；||·||_F为F范数。Among them, S is the source signal;

is the maximum a posteriori estimate of the source signal; p(S|B) is the posterior distribution of the source signal S; p(S) is the prior distribution of the source signal; L is the lead matrix; B is the EEG signal on the surface of the brain scalp data; λ is the regular parameter, I is the identity matrix; ||·|| _F is the F norm.

Further, using Bayesian probability inference, the regularization parameter λ is automatically learned in a data-driven manner, and the parameter is expressed as:

λ ^-1 = γ

其中∑_S为源信号高斯分布的方差；∑_ε表示观测噪声高斯分布的方差。优选的，迭代更新，直到p(S|B)收敛或者相对变化小于某个阈值(比如10^-6)。Among them, γ ^(k) represents the k-th iteration value; ∑ _b is the intermediate parameter, defined as

where ∑ _S is the variance of the Gaussian distribution of the source signal; ∑ _ε is the variance of the Gaussian distribution of the observation noise. Preferably, iteratively update until p(S|B) converges or the relative change is smaller than a certain threshold (such as 10 ^-6 ).

Further, using mutual information to construct a brain functional connectivity matrix includes: for each object, calculating the mutual information values of 26 spatially interesting regions respectively, and obtaining the brain functional connectivity matrix from the self-information value of the data, where the region x and the region The mutual information value of y is expressed as:

where p(x), p(y) and p(x,y) represent the x, y probability density and the joint probability density, respectively.

Further, acquiring the feature vector of the EEG data includes the following processes:

Add the covariance matrices obtained from the two categories of EEG data, arousal and valence, to obtain a regularization that conforms to the spatial covariance;

Perform white transformation on the covariance matrices of the two categories respectively, and decompose the white transformed covariance matrix to obtain the whitened space eigenvector matrix;

And according to the obtained whitening space eigenvector matrix and whitening value matrix, the projection matrix is obtained;

According to the custom feature parameter α, the projection matrix of the first α and the last α column is reserved to form the most discriminative image;

Project a trial against the most discriminative image and form the eigenvectors from the variance of the rows of the projection matrix.

Compared with the traditional CSP, which uses EEG data as input for feature extraction and classification, the present invention performs brain power imaging analysis on the preprocessed EEG signal, constructs a covariance matrix, and uses regularization technology to solve small problems. For the sample problem, in addition to the EEG signal, the EEG data of other individuals is introduced to form a formula for regularized covariance estimation, which reduces the error due to a small number of training samples; traditional CSP algorithms rely on sample-based covariance matrix estimation, However, the present invention uses the RCSP algorithm, which integrates the covariance matrix of other people's EEG samples, reduces the estimation deviation, improves the estimation stability, and improves the accuracy of the final classification result.

Description of drawings

FIG. 1 is a flowchart of a method for extracting regularized co-spatial pattern features based on brain power imaging and cortical brain network according to the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present invention proposes an emotion recognition method based on brain power imaging and regularized co-spatial pattern, which specifically includes the following steps:

Collect EEG signals and preprocess the EEG signals;

The preprocessed EEG signal is processed by the Bayesian least mode algorithm, and the neural electrical activity of the cerebral cortex is reconstructed by constructing the brain power imaging;

Through the least-modulus algorithm, the reconstructed EEG time series is projected onto the Brodman partition, which includes 26 regions of spatial interest. After flipping the source signals in opposite directions, all source signals in the 26 regions of spatial interest are The time series is averaged;

Using the mutual information of 26 spatial regions of interest to construct a brain functional connectivity matrix;

Based on the brain functional connectivity matrix, the covariance matrix of the generalized samples is constructed by regularization, and the optimal spatial filter is obtained to maximize the variance value difference between the two types of signals, so as to obtain the eigenvectors with higher discrimination;

The features of the historical data are input into the classifier for training, and the feature vector of the data to be classified is input into the classifier to obtain the EEG emotion classification.

As shown in Figure 1, the present invention is divided into training stage set data processing and test set data processing, both data need to be subjected to EEG brain power imaging and cortical ROI signal acquisition, and the training set data is subjected to RCSP spatial filter calculation and feature Extract and transfer the process to a feature extractor. The test set data obtains the RCSP features of the data through this process, trains the classifier on the features extracted from the training set data, and migrates to obtain an EEG sentiment classifier after the training is completed. , which is used to classify the RCSP features of the test set data. The above process mainly includes the following steps:

(1) Signal preprocessing, mainly denoising, in order to reduce the interference of some non-EEG signals and the effect of differences between individuals.

(2) The preprocessed EEG data is processed by the Bayesian least-modulus algorithm, and the neural electrical activity of the cerebral cortex is reconstructed by constructing brain power imaging.

(3) Through the least modulus algorithm, the reconstructed EEG time series is projected onto the Brodman partition, which includes 26 spatial regions of interest. After flipping the source signals in the opposite direction, the time series of all source signals in the ROI are taken as average value.

(4) Using mutual information to construct a brain functional connectivity matrix

(5) Using the RCSP algorithm, the estimation of the covariance matrix is regularized by two parameters, thereby reducing the estimation variance and the estimation bias at the same time.

(6) Extract the cortical-level classification features from the required feature vector, and use pattern classifiers such as SVM or KNN to achieve EEG emotion classification.

1. Source imaging based on Bayesian minimum mode solution

Using brain power imaging calculations, cortical source signals are reconstructed to obtain neural electrical activity with high temporal and spatial resolution simultaneously. The EEG potential distribution on the scalp surface of the human brain is caused by the neural current source in the brain. The propagation law of the electromagnetic field in the biological conductor satisfies the quasi-static Maxwell equations. The relationship between the EEG potential distribution on the scalp surface and the endogenous spatial signal of the human brain can be used The following linear relationship is expressed:

B=LS+ε (1)

表示在人脑头皮表面d_b个电极测量的T个采样时间点上脑电信号数据。

表示第t个采样的观测信号。

是源空间内d_s个源的源信号，

表示t时刻的皮质神经活动。是观测噪声。in,

represents the EEG signal data at T sampling time points measured by the _db electrodes on the surface of the human brain scalp.

represents the observed signal of the t-th sample.

is the source signal of d _s sources in the source space,

represents the cortical neural activity at time t. is the observation noise.

表示导联矩阵，描述特定位置和方向的源信号与头皮表面测量的脑电信号的关系，其受到电极数目、源信号数目及头模型的约束。

Represents a lead matrix that describes the relationship between source signals at a specific location and orientation and EEG signals measured on the scalp surface, constrained by the number of electrodes, the number of source signals, and the head model.

Assuming that the prior distribution of the source signal is p(S) and the observation noise ε follows the Gaussian distribution N(0,Σ _ε ), the likelihood distribution is P(B|S)～N(LS,Σ _ε ), and Σ _ε is the observation Noise covariance. Without loss of generality, the observation equation (1) is spatially whitened. Specifically, the eigenvalue decomposition of the observed noise covariance is performed to obtain:

B=LS+ε (2)

且

I为单位矩阵。为表示方便,在后文假设观测模型已白化，并去掉上式中变量符号上的波浪线。in,

and

I is the identity matrix. For convenience, in the following text, it is assumed that the observation model has been whitened, and the wavy line on the variable symbol in the above formula is removed.

Given a certain prior distribution p(S) of the source signal in the cerebral cortex, according to the Bayesian formula, the posterior distribution of the source signal S is:

则S的最大后验估计

为Using the minimum modulus solution (MNE), the source structure with the smallest energy (measured by the L2 norm) is selected as the final source signal estimate. MNE algorithm assumptions

Then the maximum a posteriori estimate of S

for

将λ的最大后验估计作为正则参数的估计值。假设λ的先验服从均匀分布，则The regularization parameter λ has an important influence on the final source signal estimation, and can generally be selected by methods such as experience or cross-validation. The system uses Bayesian probability inference to automatically learn λ in a data-driven manner. Specifically, by maximizing the posterior distribution of λ

Take the maximum a posteriori estimate of λ as an estimate of the regularization parameter. Assuming that the prior of λ obeys a uniform distribution, then

Among them, p(B|λ)=∫p(B|S)p(S|λ)dS～N(0,Σ _B ),

得到make

get

Among them, x ^(k) represents the k-th iteration value of x. Iteratively update λ until p(B|λ) converges or the relative change is less than a certain threshold (eg 10 ^-6 ).

2. ROI time series acquisition and cortical brain functional connectivity matrix calculation

ROI time series acquisition: Through the least modulus algorithm, the reconstructed EEG time series is projected onto the Brodman partition, which includes 26 regions of interest (Region Of Interest, ROI). The time series of all the source signals in the system are averaged. By extracting the brain power signal of the ROI, the spatial resolution of the EEG signal is fundamentally improved.

间的互信息为：Cortical Brain Functional Connectivity Matrix Computation: Experiments use mutual information to measure functional connectivity between electrodes. Random Variables

The mutual information between is:

where p(x), p(y) and p(x,y) represent the x, y probability density and joint probability density, respectively, and P is the vector length of the random variable. For each sample, the mutual information value of N brain intervals is calculated to obtain an N×N brain functional connectivity matrix. Taking the DEAP data set as an example, a single test sample of a certain divided time period of the subjects was selected, with a size of 26*2560, and the nchoosek function was used to select two rows of data at a time, and the probability density of each row of data and the probability density of the two rows of data were obtained respectively. The joint probability density corresponds to p(x), p(y), and p(x,y) in formula (2). According to the (joint) probability density, the mutual information MI _xy of the two rows of data can be obtained, and then cyclically obtain The obtained sample size is 26*2560 data and the symmetric brain functional connectivity matrix with 0 diagonals is 26*26.

3. Feature extraction and classification based on RCSP and cortical brain network

Covariance Matrix Estimation in CSP Method

In the process of imagining hand movement, the CSP algorithm is widely used in the processing of multi-channel EEG signals. It extracts several spatial filters so that the variance of the filtered signal is the most discriminative for both classes. In the CSP-based EEG signal classification, an EEG experiment with a number of N channels is represented by an E matrix of size N×T, and there are T samples in each channel, and each sample is treated as a separate experiment. The normalized sample covariance matrix S of each sample experiment E is:

where the superscript 'T' denotes the transpose of the matrix and tr( ) is the trace (sum of diagonal elements) of the matrix. The present invention only considers the binary class problem, so there are only two classes, and the two classes are indexed by c={1,2}. For simplicity, assume that M trials can be trained for a subject's subject in each class, m being E _(c,m) , where m=1,...,M. Therefore, each trial has a corresponding covariance matrix S _(c,m) .

The mean spatial covariance matrix for each class is then computed as:

Since there are fewer training data samples for new subjects, the EEG data of existing subjects is used through RCSP technology to improve the performance of EEG emotion recognition across subjects.

RCSP feature extraction:

The feature extraction of RCSP follows the classical CSP method. The regularized composite space covariance is formed and decomposed as follows:

Σ(β,γ)＝Σ ₁ (β,γ)+Σ ₂ (β,γ)＝UΛU ^T (10)

where U is the eigenvector matrix of the regularized composite spatial covariance, and Λ is the diagonal matrix of the corresponding eigenvalues of the regularized composite spatial covariance. The present invention adopts the convention that the eigenvalues are sorted in descending order.

Next, get the whitening transform as:

P=Λ ^-1/2 U ^T (11)

The first decomposition Σ ₁ (β,γ) of the regularized composite spatial covariance and its second decomposition Σ ₂ (β, γ) are whitened as:

Σ ₁ (β,γ)＝PΣ ₁ (β,γ)P ^T (12)

and

Σ ₂ (β,γ)＝PΣ ₂ (β,γ)P ^T (13)

Separately, then Σ ₁ (β,γ) can be decomposed into:

Σ ₁ (β,γ)=BΛ ₁ B ^T (14)

Among them, Λ ₁ is the diagonal matrix of the corresponding eigenvalues of the first decomposition of the regularized composite space covariance Σ ₁ (β, γ), Σ ₂ (β, γ) is decomposed in the same way, and decomposed into Σ ₂ (β, γ ) )=BΛ ₂ B ^T , where Λ ₂ is the diagonal eigenvalue matrix corresponding to the second decomposition Σ ₂ (β,γ) of the regularized composite space covariance, which is not repeated here. The full projection matrix is formed as:

W ₀ =B ^T P (15)

In order to obtain the most discriminative image, the first and last α columns W ₀ are reserved to form an N×Q, where Q=2α. For feature extraction, trial E is first projected as:

Z=W ^T E (16)

Then, the Q-dimensional eigenvector y is formed by the variance of the rows of Z:

是

的第q行，

是向量

的方差。where y _q is the qth component of y,

Yes

The qth row of ,

is a vector

Variance.

Classification: By using ten-fold cross-validation, the variance is reduced and the algorithm accuracy is improved. The dataset is divided into ten parts, and 9 of them are used as training data and 1 is used as test data in turn for experimentation.

In the case of limited data sets, using ten-fold cross-validation is equivalent to using the same model for different tests on a data set, but each training data set is not the same, which is equivalent to expanding the data set. If the mean effect of each model is good, it can be said that this model has a certain generalization ability to a certain extent.

Using SVM and KNN classification, where SVM is based on LIBSVM, RBF kernel function is selected, penalty factor and kernel parameters are determined by grid search on training data, and other parameters use default values. For the k value of the KNN algorithm, different k can be selected through specific applications. For example, selecting k=3 and k=5 in the DEAP data set can obtain the highest accuracy.

Using ten-fold cross-validation, the average of ten test results is taken as the classifier performance index under the cross-validation method, which effectively avoids overfitting and underfitting, and the obtained results are more reliable. Using SVM and KNN classification, where SVM is based on LIBSVM, RBF kernel function is selected, penalty factor and kernel parameters are determined by grid search on training data, and other parameters use default values. For the k value of the KNN algorithm, the experiment obtains the highest accuracy in the DEAP dataset by selecting different k, k=3 and k=5.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, and substitutions can be made in these embodiments without departing from the principle and spirit of the invention and modifications, the scope of the present invention is defined by the appended claims and their equivalents.

Claims (5)

1. The emotion recognition method based on brain power imaging and regularized co-spatial pattern, is characterized in that, comprises the following steps: Collect EEG signals and preprocess the EEG signals; The preprocessed EEG signal is processed by the Bayesian least-modulus algorithm, and the neural electrical activity of the cerebral cortex is reconstructed by constructing the brain power imaging, which includes the following steps: According to the propagation law of the electromagnetic field in the biological conductor, the linear relationship expression between the EEG potential distribution on the scalp surface and the endogenous spatial signal of the human brain was constructed; Perform spatial whitening on the obtained linear relationship expression; According to a prior distribution of the given cerebral cortex source signal and the Bayesian formula, calculate the posterior distribution of the source signal; Use the minimum modular solution to estimate the maximum a posteriori estimate of the source signal, that is, use the minimum modular solution to select the source structure with the smallest energy as the final source signal estimation, and this source signal estimation is used as the source signal of human brain imaging, then the maximum a posteriori of the source signal The estimate is expressed as:

Among them, S is the source signal;

is the maximum a posteriori estimate of the source signal; p(S|B) is the posterior distribution of the source signal S; p(S) is the prior distribution of the source signal; L is the lead matrix; B is the EEG signal on the surface of the brain scalp data; I is the identity matrix; ||·|| _F is the F norm; λ is the regular parameter, using Bayesian probability inference, the regular parameter λ is automatically learned through the data self-driven way, and the parameter is expressed as: λ ^-1 = γ

Among them, γ ^(k) represents the k-th iteration value; ∑ _b is the intermediate parameter, defined as

where ∑ _S is the variance of the Gaussian distribution of the source signal; ∑ _ε is the variance of the Gaussian distribution of the observation noise; Through the least-modulus algorithm, the reconstructed EEG time series is projected onto the Brodman partition, which includes 26 regions of spatial interest. After flipping the source signals in opposite directions, all source signals in the 26 regions of spatial interest are The time series is averaged; The brain functional connectivity matrix is constructed by using the mutual information of 26 spatial regions of interest, that is, for each object, the mutual information values of the 26 spatial regions of interest are calculated respectively, and the self-information value of the data is obtained to obtain the brain functional connectivity matrix, where The mutual information value of area x and area y is expressed as:

Among them, p(x) represents the probability density of x, p(y) represents the probability density of y, and p(x, y) represents the joint probability density of x and y; Based on the brain functional connectivity matrix, the covariance matrix of the generalized samples is constructed in a regularized way, and the optimal spatial filter is obtained to maximize the variance value difference between the two types of signals of the arousal and valence signals of the EEG data, so as to obtain the optimal spatial filter. To obtain the feature vector of high discriminative degree, obtaining the feature vector of EEG data includes the following processes: Add the covariance matrices obtained from the two categories of EEG data, arousal and valence, to obtain a regularization that conforms to the spatial covariance; Perform white transformation on the covariance matrices of the two categories respectively, and decompose the white transformed covariance matrix to obtain a whitened space eigenvector matrix; And according to the obtained whitening space eigenvector matrix and whitening value matrix, the full projection matrix is obtained; According to the self-defined characteristic parameter α, the first α column and the reciprocal α column of the full projection matrix are reserved to form the projection matrix; Project a trial according to the projection matrix, and form the eigenvectors of the variance of the rows of the projection matrix; The features of the historical data are input into the classifier for training, and the feature vector of the data to be classified is input into the classifier to obtain the EEG emotion classification. 2. The emotion recognition method based on brain power imaging and regularized co-spatial pattern according to claim 1, wherein the whitening space feature vector matrix is expressed as:

in,

is the eigenvector matrix of the whitened space covariance matrix; U is the eigenvector matrix corresponding to its eigenvalues; Σ ₁ (β,γ) is the first decomposition of the regularized composite space covariance; the superscript T represents the transpose of the matrix, Λ is the corresponding diagonal eigenvalue matrix of the regularized composite spatial covariance Σ(β,γ). 3. The emotion recognition method based on brain power imaging and regularization co-spatial pattern according to claim 1, is characterized in that, full projection matrix is expressed as:

Among them, W ₀ is the full projection matrix;

is the eigenvector matrix of the whitened space covariance matrix; U is the eigenvector matrix corresponding to its eigenvalue; the superscript T represents the transpose of the matrix. 4. The emotion recognition method based on brain power imaging and regularized co-spatial pattern according to claim 1, is characterized in that, a test is projected according to the most distinguishable image, and the projection matrix obtained is expressed as:

in,

is the projection matrix obtained by projecting a test according to the most discriminative image; W is the matrix obtained by retaining the projection matrix of the first α and the reciprocal α column; E is the N×T test vector matrix, and N represents the EEG The number of channels for the experiment, and T is the number of samples per channel. 5. The emotion recognition method based on brain power imaging and regularization co-space pattern according to claim 1, is characterized in that, the variance of the row of projection matrix is formed into eigenvector, and the qth component of this vector is expressed as:

where y _q is the qth component of y,

Yes

The qth row of ,

is a vector

The variance; Q=2α, α is a self-defined characteristic parameter.

Images