CN119970034A - Sentiment monitoring method based on online independent component analysis - Google Patents

Abstract

The invention discloses an emotion monitoring method based on online independent component analysis, which comprises the following steps of firstly filtering, removing bad conduction and removing base line operation on an electroencephalogram signal obtained from signal acquisition equipment, secondly setting a sliding window of 1s, setting the step length to be 0.5s, dividing the electroencephalogram signal, sending divided data into a generated countermeasure network model, generating artifact signals, inputting the artifact signals and the divided signals into an online independent component analysis to fully remove the artifacts, thirdly extracting space and time characteristics of the electroencephalogram signal from the processed data respectively by using a space mapping technology and a time sequence modeling mode, and fourthly carrying out depth fusion on the time and space characteristics and inputting the time and space characteristics into a trained softmax classifier for emotion monitoring. The invention carries out real-time anti-fake operation on the real-time electroencephalogram signals, and ensures that the anti-fake delay is low and the anti-fake quality is kept at a higher level.

Description

Emotion monitoring method based on online independent component analysis

Technical Field

The invention relates to the technical field of real-time electroencephalogram signal processing, in particular to an emotion monitoring method for processing signals based on an online ICA algorithm.

Background

In recent years, people pay more attention to health and emotion, and emotion monitoring technology is used as an emerging biological signal processing method and is widely studied and applied in multiple fields of psychology, medicine, man-machine interaction and the like. The identification of the emotion state not only can help people understand the emotion change of the people, but also can better understand the health condition of the people.

The brain electrical signals (EEG, electroencephalography) are used as biological signals of brain activity and can well reflect the emotional state of a person. By collecting real-time brain electrical signals of a human body and analyzing waveform characteristics, the emotion state of the human can be accurately judged. Meanwhile, there are many interferences of exogenous and endogenous signals, such as an eye electrical signal (EOG) and an electromyographic signal (EMG), etc., on the acquisition of the electroencephalogram signal. More efficient processing is required to remove these interfering signals.

Independent component analysis (Independent ComponentAnalysis, ICA) is a powerful blind source separation algorithm, and can effectively remove interference signals in brain electrical signals so as to accurately judge emotion.

However, the current ICA algorithm relies on an offline mode, and usually processes after the electroencephalogram signal is collected, so that there is a delay, and emotion monitoring cannot be achieved. In practice, the real-time monitoring can reflect the human body state more timely, so that the method can be used in places with higher emotion monitoring requirements, such as medical treatment, man-machine interaction and the like.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides the emotion monitoring method based on the online independent component analysis, which can perform real-time signal processing and emotion monitoring while acquiring the electroencephalogram signals, eliminates the delay problem of the traditional ICA processing mode, and can well remove artifacts and extract the required electroencephalogram signals by combining with the ICA algorithm after GAN optimization, thereby effectively improving the accuracy and the robustness of emotion monitoring.

In order to achieve the above purpose, the technical scheme of the invention comprises the following steps:

S1, acquiring a user real-time brain signal, and primarily preprocessing, including filtering, removing bad guides and removing base lines;

S2, segmenting the preprocessed brain signals according to a sliding window, sending the segmented data into a generated countermeasure network model, generating similar artifact signals, and carrying out independent component analysis and artifact removal together with the segmented data;

S3, extracting the spatial and temporal characteristics of the brain electrical signals from the obtained data by using a convolutional neural network and a self-attention mechanism respectively;

s4, carrying out deep fusion on the extracted space and time characteristics, and inputting the deep fusion into a softmax classifier for emotion classification, so as to realize online real-time emotion monitoring.

Preferably, step S1, the preliminary pretreatment comprises the following:

(1) Filtering by adopting a band-pass filter with the frequency of 1-45 Hz;

(2) Judging based on the amplitude and waveform stability of the brain electrical signal, and marking bad leads;

(3) Selecting a signal in a certain section of static state, calculating a baseline signal, and subtracting the baseline value;

T is the time of a certain resting state and the subsequent electroencephalogram signal is subtracted by the baseline value.

Preferably, in step S2, the size of the sliding window is set to 1S, the step size is set to 0.5S, and the segmented electroencephalogram signal:

X={x₁,x₂,...,x_t}

t represents the T-th sliding window;

Generating an artifact signal similar to the real signal in statistics through a training generator, comparing the generated artifact signal with the real EEG signal, performing the de-oculography on the signals by using ICA, performing the de-artifact processing on the myoelectric signal, and improving the quality and efficiency of de-artifact.

Preferably, in step S3, the EEG signal x= { X ₁,x₂,...,x_t }, where X _t represents the t-th sliding window, processed in step S2 is first determined according to the electrode position of the electroencephalogram signal acquisition apparatus;

Inputting the segmented electroencephalogram signals into a generating countermeasure Network model (GENERATIVE ADVERSARIAL Network), enabling a Generator (Generator) to generate artifact signals as far as possible, distinguishing the artifact signals generated by the Generator from real electroencephalogram signals by a discriminator (Discriminator), and continuously optimizing the Generator to generate the artifact signals close to the real artifact signals on the basis. And the generated artifact signals and the segmented original electroencephalogram signals are subjected to independent component analysis, so that the signals after ICA processing can effectively remove endogenous signals (such as electrooculogram, electromyogram signals and the like) and other interference signals. The independent component analysis algorithm after GAN optimization can further extract the real brain electrical signals and improve the robustness.

Preferably, in step S3, the one-dimensional electroencephalogram signal processed in step two is mapped into a two-dimensional matrix according to the electrode position of the electroencephalogram signal acquisition device:

m represents an mth electrode;

Spatial feature extraction of two-dimensional signal frame X _t using convolutional neural networks, using a 3X 3 convolutional kernel and a 2X 2 convolutional kernel, without pooling operations to preserve more spatial feature information:

F₁＝Conv(X,W₁,b₁)

F₂＝Conv(X,W₂,b₂)

W is a convolution layer, b is a bias term, and F is an output feature map;

Performing time feature extraction on the EEG signals processed in the second step by using a self-attention mechanism, and performing weighted summation to obtain final time feature vectors Ft, wherein the EEG signals are mapped to three different matrixes through different weights:

Query matrix (Q) q=x _tw_q;

a key matrix (K) k=x _tw_k;

a value matrix (V) v=x _tw_v;

Wherein w _q,w_k,w_v is a weight matrix obtained by learning;

The similarity of the sliding window, i.e. the attention score, is then calculated:

and carrying out normalization operation:

And finally, carrying out weighted summation on the output matrix V to obtain a final time feature vector:

Output=Attention Weights×V。

preferably, step S3, the first convolution kernel:

F₁＝ReLU(W₁*X(t)+b₁)

wherein, represents convolution operation, reLU is an activation function, F ₁ is a feature matrix after convolution, and b ₁ is a bias term;

A second convolution kernel:

F₂＝ReLU(W₂*F₁+b₂)

And meanwhile, the pooling operation is not used, so that the spatial characteristics are kept to the maximum extent, and finally, the extracted spatial characteristic matrix F ₂ is flattened into a one-dimensional vector F _s.

Preferably, in step S4, the spatial and temporal features extracted from CNN and the self-attention mechanism are respectively F _s and F _t, and the temporal feature F _t of the EEG signal is extracted by the self-attention mechanism, and the spatial and temporal features are spliced to form a new fusion feature vector F _f,F_f＝[F_s||F_t, ||represents the splicing operation of the features;

the fusion characteristics are expressed as:

F_f＝concat(F_s,F_t)

Finally, F _f is input into a softmax classifier for emotion classification:

θ _k is the weight of the kth emotion, k is the category number, y _k is the prediction probability of the kth emotion, and finally, the emotion with the highest prediction probability is selected as the output.

The technical principle and the beneficial effects of the invention are as follows:

The on-line ICA algorithm provided by the invention can be used for carrying out real-time signal processing and emotion monitoring while acquiring the brain electrical signals, so that the delay problem of the traditional ICA processing mode is eliminated, and meanwhile, by combining with the ICA algorithm after GAN optimization, artifacts can be well removed and the required brain electrical signals can be extracted, and the accuracy and the robustness of emotion monitoring can be effectively improved.

Drawings

FIG. 1 is a flow chart of an algorithm of the present invention;

fig. 2 is a schematic diagram of the framework of the GAN-based optimization ICA algorithm of the present invention.

Detailed Description

The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are only preferred embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Examples

As shown in fig. 1, the present invention comprises the following steps:

S1, acquiring a user real-time brain signal, and primarily preprocessing, including filtering, removing bad guides and removing base lines;

S2, segmenting the preprocessed brain signals according to a sliding window, sending the segmented data into a generated countermeasure network model, generating similar artifact signals, and carrying out independent component analysis and artifact removal together with the segmented data;

S3, extracting the spatial and temporal characteristics of the brain electrical signals from the obtained data by using a convolutional neural network and a self-attention mechanism respectively;

s4, carrying out deep fusion on the extracted space and time characteristics, and inputting the deep fusion into a softmax classifier for emotion classification, so as to realize online real-time emotion monitoring.

Preferably, step S1, the preliminary pretreatment comprises the following:

(1) Filtering by adopting a band-pass filter with the frequency of 1-45 Hz;

(2) Judging based on the amplitude and waveform stability of the brain electrical signal, and marking bad leads;

(3) Selecting a signal in a certain section of static state, calculating a baseline signal, and subtracting the baseline value;

T is the time of a certain resting state and the subsequent electroencephalogram signal is subtracted by the baseline value.

Preferably, in step S2, the size of the sliding window is set to 1S, the step size is set to 0.5S, and the segmented electroencephalogram signal:

X={x₁,x₂,...,x_t}

t represents the T-th sliding window;

Generating an artifact signal similar to the real signal in statistics through a training generator, comparing the generated artifact signal with the real EEG signal, performing the de-oculography on the signals by using ICA, performing the de-artifact processing on the myoelectric signal, and improving the quality and efficiency of de-artifact.

Preferably, in step S3, the EEG signal x= { X ₁,x₂,...,x_t }, where X _t represents the t-th sliding window, processed in step S2 is first determined according to the electrode position of the electroencephalogram signal acquisition apparatus;

Inputting the segmented electroencephalogram signals into a generating countermeasure Network model (GENERATIVE ADVERSARIAL Network), enabling a Generator (Generator) to generate artifact signals as far as possible, distinguishing the artifact signals generated by the Generator from real electroencephalogram signals by a discriminator (Discriminator), and continuously optimizing the Generator to generate the artifact signals close to the real artifact signals on the basis. And the generated artifact signals and the segmented original electroencephalogram signals are subjected to independent component analysis, so that the signals after ICA processing can effectively remove endogenous signals (such as electrooculogram, electromyogram signals and the like) and other interference signals. The independent component analysis algorithm after GAN optimization can further extract the real brain electrical signals and improve the robustness.

Preferably, in step S3, the one-dimensional electroencephalogram signal processed in step two is mapped into a two-dimensional matrix according to the electrode position of the electroencephalogram signal acquisition device:

m represents an mth electrode;

Spatial feature extraction of two-dimensional signal frame X _t using convolutional neural networks, using a 3X 3 convolutional kernel and a 2X 2 convolutional kernel, without pooling operations to preserve more spatial feature information:

F₁＝Conv(X,W₁,b₁)

F₂＝Conv(x,W₂,b₂)

W is a convolution layer, b is a bias term, and F is an output feature map;

Performing time feature extraction on the EEG signals processed in the second step by using a self-attention mechanism, and performing weighted summation to obtain a final time feature vector F _t, wherein the EEG signals are mapped to three different matrixes through different weights:

Query matrix (Q) q=x _tw_q;

a key matrix (K) k=x _tw_k;

a value matrix (V) v=x _tw_v;

Wherein w _q,w_k,w_v is a weight matrix obtained by learning;

The similarity of the sliding window, i.e. the attention score, is then calculated:

and carrying out normalization operation:

And finally, carrying out weighted summation on the output matrix V to obtain a final time feature vector:

Output=AttentionWeights×V。

preferably, step S3, the first convolution kernel:

F₁＝ReLU(W₁*X(t)+b₁)

wherein, represents convolution operation, reLU is an activation function, F ₁ is a feature matrix after convolution, and b ₁ is a bias term;

A second convolution kernel:

F₂＝ReLU(W₂*F₁+b₂)

And meanwhile, the pooling operation is not used, so that the spatial characteristics are kept to the maximum extent, and finally, the extracted spatial characteristic matrix F ₂ is flattened into a one-dimensional vector F _s.

Preferably, in step S4, the spatial and temporal features extracted from the CNN and the self-attention mechanism are respectively F _S and F _t, and the temporal feature Ft of the EEG signal is extracted by the self-attention mechanism, and the spatial and temporal features are spliced to form a new fusion feature vector F _f,F_f＝[F_s||F_t ], ||represents the splicing operation of the features;

the fusion characteristics are expressed as:

F_f＝concat(F_S,F_t)

Finally, F _f is input into a softmax classifier for emotion classification:

θ _k is the weight of the kth emotion, k is the category number, y _k is the prediction probability of the kth emotion, and finally, the emotion with the highest prediction probability is selected as the output.

Implementation flow

The emotion monitoring method based on the online ICA algorithm comprises the following specific steps:

And S1, acquiring real-time brain electrical signals and performing preliminary pretreatment. The method specifically comprises the following steps:

The acquired real-time electroencephalogram signal EEG _t is set to have a signal amplitude x _t at each time point, where t represents the time point, and further, preliminary pretreatment is required:

S11, a band-pass filter H (omega) with the frequency of 1-45 Hz is used, and the frequency response is defined as:

Y(ω)=X(ω)·H(ω)

where X (ω) is the spectrum of the original EEG signal and Y (ω) is the filtered signal spectrum.

And S12, judging based on the amplitude and waveform stability of the electroencephalogram signals, and marking bad leads. If the amplitude of the signal is greater than the threshold e for a period of time, it can be marked as bad guide:

||x(t)||_∞≥∈。

S13, baseline correction, namely collecting a baseline signal b of a user in a resting state, wherein the corrected signal is as follows:

x′_t,x'(t)=x(t)-b

Step S2, generating an antagonism network (GAN) optimization ICA algorithm, referring to a GAN-based optimization ICA algorithm framework schematic diagram in FIG. 2, specifically:

S21, in order to ensure the efficient segmentation of the real-time signal flow and the quality of the segmented signals, a sliding window mode is adopted. However, the size of the sliding window and the selection of the step size are also important, the too small window may not capture enough signal characteristics, so that the separation effect is poor, the too large window may increase the calculation load and has the problem of signal delay, the step size is too small, the calculation complexity of the algorithm is increased, and if the step size is too large, the signal segmentation is discontinuous, so that the final emotion classification is influenced.

Finally, a size of:

T_win=1s

The step length is as follows:

Δt=0.5s。

The final split signal is then:

X_t＝{x(t),x(t+1),...,x(t+T_win-1)},t∈Z

Where X _t is denoted as the signal of the t-th window.

S22, generating an countermeasure network comprises a generator G and a discriminator D, wherein the generator is used for generating an artifact signal S (t) (such as electrooculogram, electromyogram signals and the like) which is close to reality.

S(t)=G(z)

Where G (z) represents the output of the generator and z represents random noise.

The purpose of the discriminator is to distinguish the generated artifact signal from the real signal, the discriminator will make a binary decision for each signal x (t), and output a result D (x) representing the probability that the signal is a real signal.

Where W _D represents the weight of the arbiter and b _D is the bias term.

At the same time, there is a problem in that training of GAN is a challenge process involving continuous iterations of the generator and arbiter, which typically requires a lot of computational resources and time. In order to ensure the real-time performance of the algorithm, the GAN model needs to be pre-trained on a large-scale data set in advance, so that training from zero is not needed. Only a fine tuning loss function is needed on the basis of realizing signals, so that the training process of the GAN can be converged in a short time, and the following loss functions of the GAN are obtained:

L_G=-E[logD(G(z))]

L_D=-E[logD(x(t)))]-E[log(1-D(G(z)))]

And S23, in the GAN training process, the generated artifact signal S (t) and the original electroencephalogram signal are input into an ICA algorithm, and the artifact signal is removed through blind source separation, so that an optimized independent component S _optimizied (t) is obtained.

X(t)=W_optimized·S_optimizied(t)

Where W _optimized is denoted as the optimized unmixed matrix. The blind source separation process of the ICA algorithm can be effectively optimized through the countermeasure training generator and the discriminator, the artifact interference in the electroencephalogram signal is removed, and the accuracy and the robustness of emotion monitoring are improved.

S3, extracting spatial features of the electroencephalogram signals by using CNN respectively, and extracting time features of the electroencephalogram signals by using a self-attention mechanism.

S31, extracting spatial features, namely, assuming that m electrodes of equipment worn by a user acquire brain electrical signals, mapping the brain electrical signals x (t) to corresponding electrode positions to form a two-dimensional matrix:

Two convolution kernels K ₁＝3×3,K₂ =2×2 are used to extract spatial features of different scales, while no pooling layer is used in order to preserve more spatial information. With the ReLU activation function, the final feature matrix F _spatial is:

F_spatial＝ReLU(X_conv)

finally, a leveling operation is needed to be carried out to convert the two-dimensional characteristic vector into a one-dimensional vector F _spatial

F_spatial＝Flatten(F_spatial)

S32, extracting time features, namely extracting the time features by using a self-attention mechanism. Mapping the signal x (t) into a query matrix Q, a key matrix K and a value matrix V:

Q=x_tw_q

K=x_tw_k

V=x_tw_v

where w _q、w_k、w_v is a weight matrix obtained by learning in advance. The Attention Score is then calculated and normalized and weighted summed to yield the final temporal feature F _time,

F_time=Attention Weights×V

S4, carrying out depth fusion on the time and space features to obtain a final feature vector F,

F=[F_spatial||F_time]

Where || denotes the concatenation of features. Finally, inputting the feature vector F into a Softmax classifier to obtain the prediction probability P (K) of each emotion category:

Wherein θ _k is expressed as weight of kth emotion, K is the number of categories, and P (K) is the pre-prediction of kth emotion

And (3) measuring probability, and finally, selecting emotion category y _k with the highest probability as emotion monitoring results:

y_k＝arg max_kP(K)

The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.

Claims (7)

1. The emotion monitoring method based on the online independent component analysis is characterized by comprising the following steps of:

S1, acquiring a user real-time brain signal, and primarily preprocessing, including filtering, removing bad guides and removing base lines;

S2, segmenting the preprocessed brain signals according to a sliding window, sending the segmented data into a generated countermeasure network model, generating similar artifact signals, and carrying out independent component analysis and artifact removal together with the segmented data;

S3, extracting the spatial and temporal characteristics of the brain electrical signals from the obtained data by using a convolutional neural network and a self-attention mechanism respectively;

s4, carrying out deep fusion on the extracted space and time characteristics, and inputting the deep fusion into a softmax classifier for emotion classification, so as to realize online real-time emotion monitoring.

2. The emotion monitoring method based on online independent component analysis according to claim 1, characterized by:

step S1, preliminary pretreatment comprises the following steps:

(1) Filtering by adopting a band-pass filter with the frequency of 1-45 Hz;

(2) Judging based on the amplitude and waveform stability of the brain electrical signal, and marking bad leads;

(3) Selecting a signal in a certain section of static state, calculating a baseline signal, and subtracting the baseline value;

T is the time of a certain resting state and the subsequent electroencephalogram signal is subtracted by the baseline value.

3. The emotion monitoring method based on online independent component analysis according to claim 1, characterized by:

step S2, the size of the sliding window is set to be 1S, the step length is set to be 0.5S, and the segmented brain electrical signals are obtained:

X={x₁,x₂,...,x_t}

t represents the T-th sliding window;

Generating an artifact signal similar to the real signal in statistics through a training generator, comparing the generated artifact signal with the real EEG signal, performing the de-oculography on the signals by using ICA, performing the de-artifact processing on the myoelectric signal, and improving the quality and efficiency of de-artifact.

4. The emotion monitoring method based on online independent component analysis according to claim 1, characterized by:

Step S3, firstly, according to the electrode position of the electroencephalogram signal acquisition equipment, the EEG signal X= { X ₁,x₂,...,x_t }, wherein X _t represents a t-th sliding window, which is processed in the step S2;

Inputting the segmented electroencephalogram signals into a generating countermeasure Network model (GENERATIVE ADVERSARIAL Network), enabling a Generator (Generator) to generate artifact signals as far as possible, distinguishing the artifact signals generated by the Generator from real electroencephalogram signals by a discriminator (Discriminator), and continuously optimizing the Generator to generate the artifact signals close to the real artifact signals on the basis. And the generated artifact signals and the segmented original electroencephalogram signals are subjected to independent component analysis, so that the signals after ICA processing can effectively remove endogenous signals (such as electrooculogram, electromyogram signals and the like) and other interference signals. The independent component analysis algorithm after GAN optimization can further extract the real brain electrical signals and improve the robustness.

5. The emotion monitoring method based on online independent component analysis according to claim 1, characterized by:

Step S3, mapping the one-dimensional brain electrical signals processed in the step two into a two-dimensional matrix according to the electrode positions of the brain electrical signal acquisition equipment:

m represents an mth electrode;

Spatial feature extraction of two-dimensional signal frame X _t using convolutional neural networks, using a 3X 3 convolutional kernel and a 2X 2 convolutional kernel, without pooling operations to preserve more spatial feature information:

F₁＝Conv(X,W₁,b₁)

F₂＝Conv(X,W₂,b₂)

W is a convolution layer, b is a bias term, and F is an output feature map;

Performing time feature extraction on the EEG signals processed in the second step by using a self-attention mechanism, and performing weighted summation to obtain a final time feature vector F _t, wherein the EEG signals are mapped to three different matrixes through different weights:

Query matrix (Q) q=x _tw_q;

a key matrix (K) k=x _tw_k;

a value matrix (V) v=x _tw_v;

Wherein w _q,w_k,w_v is a weight matrix obtained by learning;

The similarity of the sliding window, i.e. the attention score, is then calculated:

and carrying out normalization operation:

And finally, carrying out weighted summation on the output matrix V to obtain a final time feature vector:

Output=Attention Weights×V。

6. the emotion monitoring method based on online independent component analysis according to claim 1, characterized by:

Step S3, a first convolution kernel:

F₁＝ReLU(W₁*X(t)+b₁)

wherein, represents convolution operation, reLU is an activation function, F ₁ is a feature matrix after convolution, and b ₁ is a bias term;

A second convolution kernel:

F₂＝ReLU(W₂*F₁+b₂)

And meanwhile, the pooling operation is not used, so that the spatial characteristics are kept to the maximum extent, and finally, the extracted spatial characteristic matrix F ₂ is flattened into a one-dimensional vector F _s.

7. The emotion monitoring method based on online independent component analysis according to claim 1, characterized by:

Step S4, the spatial and temporal features extracted from the CNN and the self-attention mechanism are F _s and F _t respectively, the temporal feature F _t of the EEG signal is extracted through the self-attention mechanism, the spatial and temporal features are spliced to form a new fusion feature vector F _f,F_f＝[F_s||F_t, and I represents the splicing operation of the features;

the fusion characteristics are expressed as:

F_f＝concat(F_s,F_t)

Finally, F _f is input into a softmax classifier for emotion classification:

θ _k is the weight of the kth emotion, k is the category number, y _k is the prediction probability of the kth emotion, and finally, the emotion with the highest prediction probability is selected as the output.