CN118094317A

CN118094317A - A motor imagery EEG signal classification system based on TimesNet and convolutional neural network

Info

Publication number: CN118094317A
Application number: CN202410184880.6A
Authority: CN
Inventors: 姜彬; 任福戴; 程思宇; 廖茂宇; 刘愈涵; 唐其楷; 周炳君
Original assignee: Chongqing University of Technology
Current assignee: Chongqing University of Technology
Priority date: 2024-02-19
Filing date: 2024-02-19
Publication date: 2024-05-28

Abstract

The present invention discloses a motor imagery EEG signal classification system based on TimesNet and convolutional neural network, and relates to the technical field of human physiological signal decoding. The present invention comprises an EEG signal preprocessing module, a training data preparation module, a neural network building module and a model training and testing module; the EEG signal preprocessing module is used to filter the original EEG signal in the data set; the training data preparation module is used to divide the preprocessed data into a training set, a validation set and a test set, and perform data interception, normalization and input matrix construction; the network model construction module is used to construct a convolutional neural network model. The present invention uses the TimesNet module to realize the EEG time series information representation, so that the network model can take into account the time series feature information of the EEG signal, and use the channel attention mechanism to weight the feature matrix, highlight the high contribution features, and improve the classification accuracy of the imagined motor EEG signal.

Description

Motor imagery electroencephalogram signal classification system based on TimesNet and convolutional neural network

Technical Field

The invention relates to the technical field of human physiological signal decoding, in particular to a motor imagery electroencephalogram signal classification system based on TimesNet and a convolutional neural network.

Background

Electroencephalogram detection, which is commonly referred to by people, is the process of observing brain wave activity through electrodes placed on the scalp according to certain rules. Electroencephalogram is the overall reflection of brain nerve cell electrophysiological activity on the surface of the cerebral cortex or scalp. Brain electrical signals are generated by thousands of neurons of the brain issuing electrical signals simultaneously. The electrical activity between these neurons produces weak currents that can be captured by the electrodes.

Wherein the brain electrical activity has more different brain activity states and behavioral functions or pathological differences are divided into different frequency bands:

Delta rhythm is mainly located in the frequency range of 0.5Hz to 3.5Hz, which is characteristic of the deep sleep stage when the person is in infancy or is under the treatment of immature development and the adult is in extreme fatigue and coma or anesthesia.

The θ rhythm is mainly in the 3.5 Hz-7.5 Hz frequency band, and this wave is extremely prominent in adult physician frustration or depression and psychotic patients, and is thought to be associated with different brain activity disturbances, but is a major component in the electroencephalogram of teenagers, which strengthen during sleep, playing an important role in the brain electrical activity of infants and children.

The alpha rhythm is mainly located in the frequency range of 7.5 Hz-12.5 Hz, and is the basic rhythm of normal human brain waves, and the frequency is quite constant if no external stimulus is applied.

The beta rhythm is mainly located in the frequency range of 12.5 Hz-30.5 Hz, the wave appears when the nerve is stressed and the emotion is excited or is in high, when the person wakes up from nightmare, the original slow wave rhythm can be replaced by the rhythm immediately, and the central area and the forehead area are the most obvious.

The characteristics of these frequency bands may be associated with different cognitive, emotional and brain functional states, but specific associations and explanations are still under investigation. The extraction method of the brain electrical signal comprises the following steps: preparing, namely determining electrode arrangement and sampling rate; signal acquisition, namely, signal acquisition, connecting electrodes and acquiring signals by using an electroencephalogram amplifier; signal preprocessing, filtering processing and artifact removal; signal analysis, time domain analysis, frequency domain analysis and time-frequency domain analysis; data interpretation and application, event-related potential and spectrum analysis, and brain-computer interface application.

The brain-computer interface originated in the seventies of the last century, and early BCI was mainly used for medical services, and was generally designed for critical patients with nerve or muscle disabilities, such as brain-controlled wheelchairs, brain-controlled text input devices, brain-controlled prostheses and mechanical arms, and so forth. With the development of artificial intelligence technology, deep learning is widely applied in industries such as bioinformatics, engineering, intelligent manufacturing and the like. In daily life, machine equipment is also gradually a family and a partner in life, has skills for executing daily operation tasks, and can complete basic interaction with robots through instruction control. The improvement of the computing capability of the computer promotes the development of deep learning to a great extent, so that various man-machine interaction modes, such as computer vision, natural voice, sensor control and physiological signal control, appear.

The brain electrical signal contains abundant physiological information about nerve cells, clinically presents disease physiological information for doctors to analyze, can provide the basis for diagnosing certain brain diseases, and simultaneously provides a safe and effective treatment means for certain brain diseases. The traditional Chinese medicine composition has wide clinical application, and is an important auxiliary means for diagnosing psychosis, epilepsy, brain trauma, sleep disorder, early treatment of encephalitis and the like clinically. In the application aspect of the biological signal control field, people can realize brain-computer interface technology (BCI) through brain electrical signals, and the brain-computer interface technology is used as a medium to realize a certain brain movement control purpose by utilizing brain electrical differences of human brain to sensing, movement or cognitive behaviors and operating exoskeleton machinery or intelligent equipment. For cost and portability reasons, such BCIs typically used to acquire brain electrical signals using non-invasive methods

Imagination exercise is a common electroencephalogram research paradigm, and the physiological basis is that limb exercise of a person can induce energy change of exercise rhythms in brain sense exercise areas, the phenomenon can occur in actual exercise, and a tested person with normal exercise function can also generate the phenomenon in the process of imagination exercise.

The conventional method of motor imagery classification mainly includes a co-space mode CSP method. The co-space mode is an algorithm for extracting spatial filtering characteristics under two classification tasks, and can extract spatial distribution components of each class from multi-channel brain-computer interface data. The basic principle of the common space mode algorithm is to find a group of optimal space filters for projection by utilizing diagonalization of a matrix, so that variance value difference of two types of signals is maximized, and thus, a feature vector with higher distinction degree is obtained. CSP ignores video features and focuses only on spatial features, resulting in susceptibility to non-stationarity effects of electroencephalogram signals, and easy overfitting on small datasets.

In recent years, development of artificial intelligence technology has driven research in various disciplines of deep learning. The signal-to-noise ratio of EEG is low because the measurement of brain electrical signals is often disturbed by sources of specific noise such as multiple environments, physiology and devices to mask the noise with a higher complexity, known as 'artifacts'. To overcome the above difficulties, deep learning methods are naturally applied to the processing of EEG, which can achieve better performance in task by allowing automatic end-to-end learning preprocessing, feature extraction and classification components. Among them are the representative methods of deep learning, ATCNet, proposed by Altaheri et al in "Physics-Informed Attention Temporal Convolutional Network for EEG-Based Motor Imagery Classification.", and EEGNet, proposed by LawhernVernon et al in EEGNet: a compact convolutional neural network for EEG-based brain-computur interfaces. The two methods adopt time convolution and space convolution, and after the characteristics are obtained through convolution operation, the characteristics are processed by a processing unit and then sent into a convolution classifier to realize classification. Because the existing neural network method uses a single time convolution kernel in the time domain convolution, and because of the limitation of the one-dimensional convolution kernel of the time convolution module, modeling can only be carried out between adjacent time points for a time sequence, and long-term dependence cannot be realized;

there is therefore a need to propose a new solution to the above problems.

Disclosure of Invention

The invention aims to provide a motor imagery electroencephalogram signal classification system based on TimesNet and a convolutional neural network, which utilizes a TimesNet module to realize electroencephalogram sequence information representation, so that a network model can consider sequence characteristic information of electroencephalogram signals, a channel attention mechanism is used for weighting a characteristic matrix, high contribution degree characteristics are highlighted, and classification accuracy of the imagery motor electroencephalogram signals is improved.

In order to achieve the above purpose, the present invention provides the following technical solutions: a motor imagery electroencephalogram signal classification system based on TimesNet and a convolutional neural network comprises an electroencephalogram signal preprocessing module, a training data preparation module, a neural network building module and a model training and testing module;

The electroencephalogram signal preprocessing module is used for carrying out filtering processing on original electroencephalogram signals in a data set, and a band-pass filter with band-pass filtering frequency of 1 Hz-40 Hz is used for eliminating high-frequency noise and low-frequency noise in the original electroencephalogram signals;

The training data preparation module is used for dividing the preprocessed data into a training set, a verification set and a test set, wherein 40% of the preprocessed data is divided into the training set, 10% of the preprocessed data is divided into the verification set, 50% of the preprocessed data is divided into the test set, and data interception, normalization and construction of an input matrix are carried out;

the network model construction module is used for constructing a convolutional neural network model;

The model training, verifying and testing module is used for inputting the data processed by the training data preparation module into the convolutional neural network for training and parameter updating, and completing performance testing by using the testing set.

Preferably, the convolutional neural network model is used for extracting and classifying characteristics of an input three-dimensional brain electrical signal matrix;

the convolutional neural network model comprises a feature extraction module and a fully-connected classification module;

The characteristic extraction module is used for automatically extracting characteristics of the input three-dimensional brain electrical signals through a convolutional neural network.

The fully-connected classification module is used for classifying and judging the motor imagery category corresponding to the input three-dimensional brain electrical matrix of the convolutional neural network model.

Preferably, the feature extraction module comprises TimesNet module, space-time convolution module and channel attention module;

the TimesNet module is used for realizing better characteristic characterization on the input matrix;

the space-time convolution module is used for extracting the time characteristics and the space characteristics of the output characteristics of the TimesNet module;

the channel attention module is used for distributing weights to all channels of the output characteristics of the space-time convolution module.

Preferably, the TimesNet module converts the one-dimensional time sequence feature into a two-dimensional tensor for analysis to obtain a better time sequence representation, the TimesNet module comprises a TimesBlock module, timesBlock module for extracting period of input one-dimensional time sequence data, and a one-dimensional time sequence with a time length of T and a channel dimension of CThe periodicity can be calculated from the fast fourier transform (Fast Fourier Transform, FFT) of the time dimension, namely:

A＝Avg(Amp(FFT(X_1D))),

Wherein, The intensity of each frequency distribution in X _1D is represented, the k frequencies { f ₁,…,f_k } with the greatest intensity corresponding to the most significant k period lengths { p ₁,…,p_k }. We have the above process abbreviated as:

A,{f₁,…,f_k},{p₁,…,p₂}＝Period(X_1D)

Next, it is converted into a two-dimensional tensor to represent a two-dimensional time sequence variation, we can stack the original sequence based on the selected period, and this process can be formulated as:

Where Padding () is appended 0 at the end so that the sequence length can be divided by p _i. By the above operation we obtain a set of two-dimensional tensors Corresponding to the two-dimensional timing variation dominated by period p _i.

For two-dimensional tensors, we can extract two-dimensional time-series variation characterization using classical Inception model, namely:

for extracted timing features, we convert them into one-dimensional space for information aggregation:

Finally, we calculate the one-dimensional representation and the intensity of the corresponding frequency through Softmax, and then carry out weighted summation to obtain the final output:

through the process, timesNet is completed, two-dimensional time sequence changes are respectively extracted from a plurality of periods, and then a time sequence change modeling process of self-adaptive fusion is performed.

Preferably, the space-time convolution module obtains the characteristics of the electroencephalogram signal by passing the output characteristics of the TimesNet module through a time convolution layer and a space convolution layer, and the application of the space-time convolution module at least comprises the following steps:

The time-space convolution module is sequentially connected with the time convolution layer, the space convolution layer and the activation function, wherein the time convolution layer is provided with convolution kernels with the selection size of 1 x 125, and the number of the convolution kernels is 12;

In the space convolution layer, the size of the space convolution kernel is set to be C1, the number of the space convolution kernel is equal to the number of channels of the electroencephalogram data, and the number of the space convolution kernel is set to be twice the number of the convolution kernel of the time convolution layer;

And adding a square activation function after the time convolution layer and the space convolution layer, and then connecting an average pooling layer in series, wherein the convolution kernel of the layer is 1 x 37, and then adding a nonlinear factor by utilizing a logarithmic activation function to enhance the expression capacity of the neural network to the model, wherein the convolution module is used for normalizing in batches finally, so that the training process of the network is accelerated, and the accuracy of the model is improved.

Preferably, the channel attention module averages the values of each characteristic channel through a global average pooling layer by the output characteristics of the space-time convolution module to generate a vector with the same channel number, then captures local cross-channel interaction information for each channel and their neighbors by using one-dimensional convolution operation, multiplies each component of the weight sequence by corresponding channel original data to obtain a characteristic diagram after the channel attention mechanism is acted, and the characteristic diagram is used as the output characteristics of the characteristic extraction module.

Preferably, the feature vector output by the fully-connected classification module is processed by a softmax function to obtain the score belonging to each classification.

Preferably, in the model training and testing module, the electroencephalogram data of each tested person is independently trained and tested.

Compared with the prior art, the invention has the beneficial effects that:

1. according to the invention, by using TimesNet modules for obtaining better electroencephalogram signal characterization and introducing a space-time convolution module for extracting electroencephalogram time local features and space (i.e. channel) global features, the quality of the features extracted by the network model is ensured.

2. According to the invention, through the TimesNet module, one-dimensional time sequence data can be expanded to a two-dimensional space for analysis, one-dimensional time sequences are stacked based on a plurality of periods, a plurality of two-dimensional tensors can be obtained, and the columns and rows of each two-dimensional tensor respectively reflect time sequence changes in the periods and in the periods, so that the two-dimensional time sequence changes are obtained.

3. According to the invention, the contribution degree of each channel of the characteristic diagram is weighted by introducing the channel attention mechanism, so that the high contribution degree characteristic is strengthened, the low contribution degree characteristic is weakened, the capturing capability of the network model to the effective characteristic is improved, and the classification accuracy of the network is further improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a system flow diagram of the present invention;

FIG. 2 is a diagram of a neural network model in accordance with the present invention;

FIG. 3 is a block diagram of TimesNet of the present invention;

FIG. 4 is a diagram of the ECA-Net attention module of the present invention;

FIG. 5 is a histogram of accuracy at BCICIV a for various methods of the present invention;

FIG. 6 is a histogram of accuracy of the results of the different methods of the present invention performed on the average of BCICIV b dataset;

FIG. 7 is a schematic diagram of an ablation experiment of TimesNet modules and ECANet structures in accordance with the present invention;

FIG. 8 is a visual representation of the output characteristics of subject S2 in BCICIV a during the training phase and the testing phase with the addition of TimesNet module and the removal of TimesNet module;

FIG. 9 is a schematic diagram of visualization of spatial information of an electroencephalogram data input network through a convolution module;

FIG. 10 is a time domain activation graph of an electroencephalogram data network through a spatial convolution layer;

FIG. 11 is a class activation diagram showing the phenomenon of ERD/ERS as it occurs when left and right hand motions are envisioned.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments.

Referring to fig. 1-4, a motor imagery electroencephalogram classification system based on TimesNet and convolutional neural networks.

The present example uses BCI-Competition-IV-2a to disclose an electroencephalogram dataset of the dataset, which data total 25 channels, of which 22 channels are EEG and 3 channels are EOG, of which the data of the three channels of EOG do not participate in classification, the data is sampled using 250Hz, 4 motor imagery classifications of 9 subjects are collected, two trials are performed, each trial is repeated 72 times, two groups of trials total 576 trials are performed each time, 8s are performed.

The electroencephalogram signal preprocessing module is used for performing operations such as filtering on original electroencephalogram signals in the data set;

In the electroencephalogram signal preprocessing module, filtering means that the original electroencephalogram signal of the BCI-Competition-IV-2a dataset is subjected to band-pass filtering, and a band-pass filter with the band-pass filtering frequency of 1-40 Hz eliminates high-frequency and low-frequency noise in the original electroencephalogram signal;

the training data preparation module is used for dividing the preprocessed electroencephalogram data into a training set, a verification set and a test set, normalizing, enhancing data, constructing an input matrix and the like;

firstly, the brain electrical signal of the BCI-Competition-IV-2a dataset processed by the brain electrical signal preprocessing module is subjected to two groups of experiments of 4 motor imagination of 9 testers in total, wherein the first group of experiments are selected as a training set and a verification set, the second group of experiments are selected as a test set, standard deviation is quoted for the training set for normalization, and the mean value and variance of the training set are used for carrying out the same processing on the verification set and the test set, so that the verification set and the test set meet the probability distribution of the training set data, the dimension difference of each channel of the brain electrical data is eliminated, and the network convergence is more stable during training; then cutting data of the training set, the verification set and the test set, intercepting the time of the imagination movement process of 3 s-7 s from the time of 8s, and finally obtaining a single sample with the size of 22 x 1000; finally, the obtained electroencephalogram signals with the size of 22 x 1000 are expanded to the size of 1 x 22 x 1000 to construct a three-dimensional myoelectric matrix, and the three-dimensional myoelectric matrix is input into a model training and testing module for training.

The network model building module is used for building a convolutional neural network model;

the convolutional neural network (the structure of which is shown in fig. 2) is used for extracting and classifying the characteristics of the input three-dimensional electroencephalogram matrix, and comprises a special TimesNet module, a space-time characteristic extraction module and a fully-connected classification module.

TimesNet the module comprises a TimesBlock module, timesBlock extracting period of inputted one-dimensional time sequence data, and for one-dimensional time sequence with time length T and channel dimension CThe periodicity can be calculated from the fast fourier transform (Fast Fourier Transform, FFT) of the time dimension, namely:

A＝Avg(FFT(X_1D))),

{f₁,…,f₂}＝argTopk(A),

A,{f₁,…,f_k},{p₁,…,p₂}＝Preiod(X_1D)

Through the above process, timesNet is completed, two-dimensional time sequence changes are extracted respectively in a plurality of periods, and then a time sequence change modeling process of self-adaptive fusion is performed, and fig. 3 is a block diagram of the module.

The space-time convolution module comprises a time convolution layer, a space convolution layer and an activation function which are sequentially connected, wherein the time convolution layer is provided with convolution kernels with the selection size of 1 x 125, and the number of the convolution kernels is 12; in the space convolution layer, the size of the space convolution kernel is set to be C1, the number of the space convolution kernel is equal to the number of channels of the electroencephalogram data, and the number of the space convolution kernel is set to be twice the number of the convolution kernel of the time convolution layer; adding a square activation function after a time convolution layer and a space convolution layer, then connecting an average pooling layer in series, wherein the size of a convolution kernel of the layer is 1 x 37, then adding a nonlinear factor by utilizing a logarithmic activation function, enhancing the expression capacity of a neural network to a model, and finally carrying out batch normalization on a convolution module to accelerate the training process of the network and improve the precision of the model;

the channel attention module is used for distributing weights of channels of the output characteristics of the space-time convolution module, and fig. 4 is a structure diagram of the module;

Firstly, processing output characteristics of a multi-scale fusion convolution module through a global average pooling layer to generate a one-dimensional sequence with the length of C2; then, a one-dimensional convolution with a kernel size of 15 (padding size of 7) is used to learn the weight of each channel on a one-dimensional sequence; finally, the learned weight sequence is modified by a sigmoid activation function to produce a final weight sequence. Each component of the final weight sequence is multiplied by the corresponding raw data for each channel to generate a feature map that is processed by the channel attention mechanism, which is used as an output feature of the feature extraction module. The channel attention technology enables the network model to automatically score the contribution of each channel in the feature map, and then weights the features according to the scores so as to improve the capability of the network model for capturing effective features;

The fully-connected classification module is used for carrying out feature map dimension reduction on the output features of the channel attention module, firstly flattening the output features (the size is 24 x 1 x 43) of the attention module into one-dimensional features with the length of 1032, and sending the one-dimensional features into the fully-connected network. The method comprises the steps of sequentially performing full-connection layers (128 neurons), nonlinear activation layers and random inactivation layers (the ratio is 0.5) in a full-connection network, outputting feature vectors with the length of 4 through the full-connection layers (4 neurons), inputting the output feature vectors into a Softmax layer, and processing the feature vectors by using a Softmax function to obtain vectors with the length of 4, wherein each class of the vectors is a score belonging to each imagination movement, and corresponds to 4 classes in a BCI-Competition-IV-2a dataset.

Batch normalization in TimesNet module uses LayerNormalization, batch normalization in spatio-temporal convolution module and fully connected module uses BatchNormalization to speed up network convergence; the non-linear activation layers all use a ReLU activation function to enhance the non-linear expression capabilities of the network.

The model training and testing module is used for inputting the data processed by the training data preparation module into the convolutional neural network model for training and parameter tuning, and completing performance testing by using the testing set.

The model training and testing module is used for training and testing the electroencephalogram data of 9 testers independently, training 1000 iteration cycles are carried out on each tester, a training set is divided into a plurality of latches with the size of 64 in each iteration cycle, each latch is sequentially input into a lightweight convolutional neural network to obtain a classification result, the classification result is compared with a label, a cross entropy loss function is used for calculating loss, then an Adam optimizer is used for carrying out back propagation to update trainable parameters in the network, the initial learning rate is determined to be 0.0002, and an automatic learning rate adjustment strategy of cosine annealing is adopted. And in each iteration period, after the data of the training set are completely trained, inputting the verification set into the convolutional neural network to calculate the classification accuracy, adjusting the super-parameters of the network to ensure that the loss of the model is minimum, storing the model parameters at the moment, and finally testing the parameters with the minimum loss of the model by using a test set, wherein the accuracy result on the test set is the classification accuracy of the motor imagery electroencephalogram classification system based on the convolutional neural network.

Experimental results

1. Comparative test

To verify the superiority of the EEGTimes-ECA-Net algorithm herein in re-MI-EEG signal feature recognition, we have also conducted extensive trial-related experiments, and we selected some excellent algorithm as the baseline algorithm to compare on both public data sets. The Baseline (Baseline) algorithm is described as follows:

FBCSP ^[1]: FBCSP is that the spatial extraction method based on CSP is implemented by slicing the frequency band and adding a feature selection algorithm, specifically, firstly slicing the frequency band, then CSP filtering each sub-frequency band, and finally selecting features of the filtered features and classifying the results.

EEGNet ^[2]: a compact convolutional neural network that utilizes a convolutional kernel to learn a temporal filter to capture motion-related frequency information, and a separable convolutional network to learn a spatial filter, including deep convolution (DEPTHWISE CONVOLUTION) and point convolution (Pointwise Convolution), reduces training parameters of the model.

ShallowConvNet ^[3]: the data transformation performed by ShallowConvNet is similar to the transformation of FBCSP, with the time-rolling and spatial filters followed by a squared non-linear, mean-pooling layer and logarithmic activation function to extract deep features.

EEG-TCNet ^[4]: EEG-TCNet was proposed as an extension of EEGNet, applying the TCN structure after EEGNet feature extraction to further extract time information.

EEG-ITNet ^[5]: the model uses Inception modules and causal convolution with dilation to extract rich spectral, spatial and temporal information from the multi-channel brain electrical signal.

MBSTCNN-ECA-LightGBM ^[6]: this is a multi-branch convolutional network model with channel attention and LightGBM, which is built to learn time-frequency domain features, then add channel attention mechanisms to get more features, and finally decode classification tasks using lightweight structure LightGBM.

C2CM ^[7]: the model is classified with the CNN architecture by modifying the filter bank co-spatial pattern to generate a new data temporal representation, and the CNN is optimized for the new representation.

EEGConformer ^[8]: this is a compact transducer structure, encapsulating both local and global features extracted from the data in the same EEG classification framework. The method comprises the steps of learning local features of one-dimensional time and space by using a convolution module, and then connecting a self-attention module to extract global correlation in the local time features.

LMDANet ^[9]: the model combines two attention modules specially designed for electroencephalogram signals, and can integrate characteristics of multiple dimensions.

EEG-CDILNet ^[10]: the model provides a cyclic expansion convolution network (CDIL), which is a symmetrical structure, different from TCN, and can be used for classifying ultra-long sequence data by utilizing the expansion convolution multiplied expansion receptive field.

1.2 Results on BCICIV a

We compared the proposed model with the most advanced MIEEG classification methods using multiple indices, including classification accuracy, cohen score, classification accuracy, recall, F1 score.

Table 1 shows the classification accuracy and Kappa score for each subject for the most advanced deep-learning MI-EEG classification method in the set, using individual-specific training methods .EEGNet,ShallowConvNet,EEG-TCNet,EEG-ITNet,MBSTCNN-ECA-LightGBM,C2CM,EEGConformer,LMDANet,EEG-DCILNet and EEGTimes-ECA-Net as comparative methods on the data set. The accuracy of the proposed model EEGTimes-ECA-Net was 80.45% and fixed parameters and super parameters were used for all individuals. Firstly, the accuracy of the structure of the device is obviously improved by 12.7 percent compared with the traditional FBCSP, and the result also shows that the performances of other deep learning methods such as ShallowConvNet, EEGNet and the like are also better than FBCSP, which indicates that the CNN-based method has strong characteristic characterization capability, but has limited receptive field, and the methods only focus on local characteristics and neglect global correlation, which possibly can be related to the decoding accuracy of the electroencephalogram signal sequence; recently proposed EEGConformer introduces a transducer structure to obtain global dependency information of the electroencephalogram signals, but the attention mechanism has difficulty in finding reliable dependency from discrete time points. The C2CM effectively combines the traditional artificial feature and deep learning ideas, but although the model parameter of each tested is finely tuned, the model parameter of each tested model is still unable to defeat us except for the tested model 7, and the accuracy of the network is 6% higher than that of the MBSTCNN-ECA-LightGBM network which also uses ECANet modules, so that the model can prove that the characteristic extraction capability of the module is stronger than that of a multi-branch network, and the model has the characteristic extraction performance of a non-input multi-branch structure.

Furthermore, the proposed model accuracy is at least 1.8% higher than the attention mechanism based network EEGConformer. This means that the TimesNet structure in the system is more suitable than the transducer structure for extracting long-term correlation features of the timing signal and the model structure is simpler. Table 2 summarizes the accuracy and recall of the proposed model for each class, and we also provide an average of accuracy, recall and F1 score. See fig. 5.

Table 1 comparison between methods on BCICIV a dataset

Methods	S01S02S03S04S05S06S07S08S09Avg.(kappa)
		FBCSP[1]	76.0056.5081.2561.0055.0045.2582.7581.2570.7567.75(0.57)
EEGNet[2]	84.3454.0687.5463.5967.3954.8888.8076.7574.2472.40(0.63)
		ShallowConvNet[3]	79.5156.2588.8980.9057.2953.8291.6781.2579.1774.31(0.66)
EEG-TCNet[4]	85.7765.0294.5164.9175.0061.4087.3683.7678.0377.35(0.71)
		EEG-ITNet[5]	84.3862.8589.9369.1074.3157.6488.5483.6880.2176.74
MBSTCNN-ECA-LightGBM[6]	82.0061.0089.0063.0071.0064.072.0079.0084.0074.00(0.65)
		C2CM[7]	87.5065.2890.2866.6762.5045.4989.5883.3379.5174.46(0.66)
EEGConformer[8]	88.1961.4693.4078.1352.0865.2892.3688.1988.8978.66(0.71)
		LMDANet[9]	86.5067.4091.7077.4065.6061.1091.3083.385.4078.80(0.71)
EEG-CDILNet[10]	85.4061.8094.4073.6066.0061.8093.1091.7088.9079.63(0.72)
		EEGTimes-ECA-Net	85.3171.8891.2578.7575.0063.1388.4487.8182.5080.45(0.74)

Meter 2Precision recall and F1 Score on the BCICIV2a dataset using EEGTimesNet

1.2 Comparison on BCICIV b

BCICIV2b is a data set with low complexity of three-channel two-classification, and the model proposed by us achieves the highest classification accuracy. Our model also achieved the best effect on this dataset, while we also compared some methods. Unlike BCICIV a, FBCSP achieves higher accuracy on the BCICIV b dataset, and the deep learning method EEGNet, shallowConvNet and the like also exhibit higher classification performance, which also shows that the deep learning method has stronger feature extraction capability than the conventional method. The EEGTimes-ECA-Net performed best in predicting volatility beyond LMDANet and EEGConformer, with average accuracy of the model exceeding 86% (0.74 kappa). In the case of a small number of channels, the conventional method such as FBCSP is not advantageous, possibly because it is difficult for the spatial filter to learn enough spatial characteristics. Deep learning models such as EEGConformer with excessive parameter amounts also exhibit limitations in classification performance. Again we do not specifically optimize the hyper parameters of EEGTimes-ECA-Net for BCICIV b. The classification performance in BCICIV b reflects good robustness of LMDA-Net. See fig. 6.

Table 3 method comparisons on BCICIV b dataset

Meter 4Precision recall and F1 Score on the BCICIV2b dataset using EEGTimesNet

2. Ablation experiments

Referring to FIG. 7, no ECANet represents EEGTimes-ECA-Net without ECA attention channel mechanism;

no TimesNet denotes EEGTimes-ECA-Net without TimesNet structure.

Key to the improvement of EEGTimes-ECA-Net over CNN-based approaches is the addition of TimesNet-based modules to learn the global representation. At the same time, channel attention mechanisms may also contribute to the final result. Thus, we performed ablation experiments in the dataset described above to further verify the effectiveness of TimesNet and ECANet modules in EEGTimes-ECA-Net, we removed TimesNet and ECANet modules in turn, and compared them to a reference network. It can be seen that after removal of TimesNet modules, network performance had degraded to some extent on both datasets and on BCICIV a dataset, test 7 was degraded by 3.12%, test 8 was similarly degraded by 3.12% and the average accuracy was degraded by 4.45% (p < 0.01). The overall performance average accuracy is improved by 2.99% (p < 0.01) compared to the case without ECANet modules. On BCICIV b dataset, the drop in amplitude was small after removal of TimesNet module, test 2 dropped by 1.56% accuracy, test 8 dropped by 2.19% accuracy, and the average accuracy dropped by 1.37% (p < 0.01). Notably, in BCICIV a dataset, classification accuracy was reduced for better discrimination test 1 after addition of TimesNet module, while improvement was significant reaching 17.81% and 32.82% for test 2 and test 5, which were otherwise poorly performing. The fact that the TimesNet module adopted by the method can acquire the characteristics of the electroencephalogram time sequence is also proved, and the robustness and the characteristic extraction capability of the model are enhanced.

3. Visualization of

Visualization of output characteristics of subject S2 in fig. 8BCICIV a during the training phase and the testing phase with the addition of TimesNet module and the removal of TimesNet module. The first row represents feature visualizations of the non-added and added modules during the training phase, and the second row represents feature visualizations under both conditions during the testing phase.

Fig. 9 visualization of spatial information of an electroencephalogram data input network through a convolution module. The Without data shows that Without TimesBlock, most of the brain motor sensory area was activated and the network model was able to capture throughout the experiment; the second line shows that with the TimesBlock modules, our model can focus more on the motor sensory area based on the original focus, which is reflected in the effect of darker color in the motion area and lighter color in the other areas.

Fig. 10 is a time domain activation graph of an electroencephalogram data network through a spatial convolution layer. The data show that we have different concerns over different ranges of the time domain throughout the experiment.

The class 11 activation diagram shows the ERD/ERS phenomenon that occurs when left and right hand motions are envisioned.

The visualization method should preserve the global and local features of the data, so we further illustrate TimesNet's interpretability through two different angles, including deep feature Distribution (Distribution) of UMAP and the global dependency Representation (presentation) implied by the class activation graph.

(1) Characteristic distribution: in order to preliminarily explain the influence of the features extracted by the TimesBlock module introduced into the model on the classification performance, a new statistical degradation and visual chemistry technology is used, UMAP ^[12], and compared with the t-SNE algorithm, the UMAP algorithm reserves more global structures and more sufficient features in the aspect of visual quality. After training adequately in two ways, including TimesNet and TimesNet, we choose BCICIV a dataset to test 2 with the feature distribution shown in the figure, we can see that for training data, the distances between different classes are close without TimesNet modules, while the intra-class spacing of the features extracted by TimesNet modules is smaller and the inter-class spacing is larger; for the test data, the distance between the features in the class is large and the aliasing between the classes is very obvious without TimesNet, while with the help of TimesNet, the aliasing between the classes is improved greatly, although the class boundaries are not clear enough, considering the poor resolution of the data of the tested 2.

(2) The characteristic is represented as follows: to verify that the proposed model was able to achieve a better representation of the electroencephalogram features by introducing TimesNet modules, we next verified the validity of the modules by further more intuitive visualizations, whereupon we used gradient weighted class activation mapping (gradCAM) to demonstrate that our model learns the global feature representation through the BCICIV a dataset as shown in fig. 9, which represents a separate representation of the average of all training trial experiments for each subject. We use class activation graphs (CAMs) to monitor the network's interest in different areas of time period as shown in fig. 10, where we have only chosen the average sample data for the first second of each test due to space constraints. It can be seen that different attentiveness was presented in different time domains, indicating that the test was attentive due to fatigue during the course of the experiment and that the motor awareness was presenting a certain delay. Next, by looking at each of the left and right hand trial class activation charts of fig. 11, it is evident that event-related synchronization and de-synchronization, and that apparent contralateral activation and ipsilateral inhibition were observed in trial 1,3,7,8 et al. However, the event correlation characteristics of the two persons tested 2 and 6 are not obvious, and the classification effect of the two persons is poor.

Reference is made to:

[1]KAI KENG A,ZHENG YANG C,HAIHONG Z,et al.Filter Bank Common Spatial Pattern(FBCSP)in Brain-Computer Interface[C]//Filter Bank Common Spatial Pattern(FBCSP)in Brain-Computer Interface.2008IEEE International Joint Conference on Neural Networks(IEEE World Congress on Computational Intelligence),1-8June 2008.2390-2397.

[2]LAWHERN V J,SOLON A J,WAYTOWICH N R,et al.2018.EEGNet:a compact convolutional neural network for EEG-based brain-computer interfaces[J].JNeural Eng,15(5):056013.

[3]SCHIRRMEISTER R T,SPRINGENBERG J T,FIEDERER L D J,et al.2017.Deep learning with convolutional neural networks for EEG decoding and visualization[J].Hum Brain Mapp,38(11):5391-5420.

[4]INGOLFSSON T,HERSCHE M,WANG X,et al.EEG-TCNet:An Accurate Temporal Convolutional Network for Embedded Motor-Imagery Brain–Machine Interfaces[M].City,2020.

[5]SALAMI A,ANDREU-PEREZ J,GILLMEISTER H.2022.EEG-ITNet:An Explainable Inception Temporal Convolutional Network for Motor Imagery Classification[J].IeeeAccess,10:36672-36685.

[6]JIA H,YU S,YIN S,et al.2023.A Model Combining Multi Branch Spectral-Temporal CNN,Efficient Channel Attention,and LightGBM For MI-BCI Classification[J].IEEE Trans Neural Syst Rehabil Eng,PP:1311-1320.

[7]WANG G,CERF M.2022.Brain-Computer Interface using neural network and temporal-spectral features[J].FrontNeuroinform,16:952474.

[8]SONG Y,ZHENG Q,LIU B,et al.2022.EEG Conformer:Convolutional Transformer for EEG Decoding and Visualization[J].IEEE Trans Neural Syst Rehabil Eng,PP:710-719.

[9]MIAO Z,ZHAO M,ZHANG X,et al.2023.LMDA-Net:A lightweight multi-dimensional attention network for general EEG-based brain-computer interfaces and interpretability[J].Neuroimage,276:120209.

[10]LIANG T,YU X,LIU X,et al.2023.EEG-CDILNet:a lightweight and accurate CNN network using circular dilated convolution for motor imagery classification[J].J Neural Eng,20(4).

[11]ZHAO H,ZHENG Q,MA K,et al.2021.Deep Representation-Based Domain Adaptation for Nonstationary EEG Classification[J].IEEE Trans Neural Netw Learn Syst,32(2):535-545.

[12]MCINNES L,HEALY J.2018.UMAP:Uniform Manifold Approximation and Projection for Dimension Reduction[J].ArXiv,abs/1802.03426.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.

Claims

1. A motor imagery EEG signal classification system based on TimesNet and convolutional neural network, characterized by: comprising an EEG signal preprocessing module, a training data preparation module, a neural network building module and a model training and testing module;

The EEG signal preprocessing module is used to filter the original EEG signals in the data set, using a bandpass filter with a bandpass filter frequency of 1 Hz-40 Hz to eliminate high-frequency and low-frequency noise in the original EEG signals;

The training data preparation module is used to divide the preprocessed data into a training set, a validation set and a test set, wherein 40% is divided into a training set, 10% is divided into a validation set, and 50% is divided into a test set, and perform data interception, normalization and input matrix construction;

The network model building module is used to build a convolutional neural network model;

The model training, verification and testing module is used to input the data processed by the training data preparation module into the convolutional neural network for training and parameter updating, and use the test set to complete performance testing.

2. The motor imagery EEG signal classification system based on TimesNet and convolutional neural network according to claim 1, characterized in that: the convolutional neural network model is used to extract features and classify the input three-dimensional EEG signal matrix;

The convolutional neural network model includes a feature extraction module and a fully connected classification module;

The feature extraction module is used to automatically extract features of the input three-dimensional EEG signal through a convolutional neural network.

The fully connected classification module is used to classify and distinguish the motor imagery category corresponding to the input three-dimensional EEG matrix of the convolutional neural network model.

3. The motor imagery EEG signal classification system based on TimesNet and convolutional neural network according to claim 2, characterized in that: the feature extraction module includes a TimesNet module, a spatiotemporal convolution module and a channel attention module;

The TimesNet module is used to achieve better feature representation of the input matrix;

The spatiotemporal convolution module is used to extract temporal features and spatial features from the output features of the TimesNet module;

The channel attention module is used to assign weights to each channel of the output features of the spatiotemporal convolution module.

4. According to claim 3, a motor imagery EEG signal classification system based on TimesNet and convolutional neural network is characterized in that: the TimesNet module converts the one-dimensional time series features into a two-dimensional tensor for analysis to obtain a better time series representation, and the TimesNet module is composed of a TimesBlock, which extracts the cycle of the input one-dimensional time series data. For a one-dimensional time series with a time length of T and a channel dimension of C Its periodicity can be calculated by the Fast Fourier Transform (FFT) of the time dimension, namely:

A＝Avg(Amp(FFT( _X1D ))),

in, represents the intensity of each frequency distribution in _X1D . The k frequencies with the largest intensity { _f1 , ..., _fk } correspond to the k most significant period lengths { _p1 , ..., _pk }. We can abbreviate the above process as:

A,{f ₁ ,…,f _k },{p ₁ ,…,p ₂ }＝Period(X _1D )

Next, convert it into a two-dimensional tensor to represent the two-dimensional time series changes. We can stack the original sequence based on the selected period. The process can be formulated as:

Among them, Padding() adds 0 at the end so that the sequence length can be divided by _pi . Through the above operations, we get a set of two-dimensional tensors It corresponds to the two-dimensional time series changes dominated by the period p _i .

For a two-dimensional tensor, it has two-dimensional locality. Therefore, we can use the classic Inception model to extract a two-dimensional temporal change representation, namely:

For the extracted time series features, we transform them into a one-dimensional space for information aggregation:

Finally, we use Softmax to calculate the one-dimensional representation and the intensity of its corresponding frequency for weighted summation to get the final output:

Through the above process, TimesNet completes the process of extracting two-dimensional time series changes in multiple cycles and then performing adaptive fusion time series change modeling.

5. The motor imagery EEG signal classification system based on TimesNet and convolutional neural network according to claim 4 is characterized in that: the spatiotemporal convolution module obtains the features of the EEG signal by using the output features of the TimesNet module through a temporal convolution layer and a spatial convolution layer, and the application of the spatiotemporal convolution module at least comprises the following steps:

The spatiotemporal convolution module sequentially connects the temporal convolution layer and the spatial convolution layer, as well as the activation function. The convolution kernel size in the temporal convolution layer is set to 1*125, and the number of convolution kernels is 12;

In the spatial convolution layer, the size of the spatial convolution kernel is set to C*1, where C is the same as the number of channels of the EEG data, and the number of spatial convolution kernels is set to twice the number of convolution kernels in the temporal convolution layer;

A square activation function is added after the temporal convolution layer and the spatial convolution layer, followed by a series of average pooling layers with a convolution kernel size of 1*37. The logarithmic activation function is then used to add nonlinear factors to enhance the neural network’s ability to express the model. The convolution module is then batch normalized to accelerate the network training process and improve the accuracy of the model.

6. According to claim 5, a motor imagery EEG signal classification system based on TimesNet and convolutional neural network is characterized in that: the channel attention module averages the output features of the spatiotemporal convolution module through a global average pooling layer to generate a vector with the same number of channels, and then uses a one-dimensional convolution operation on each channel and their neighbors to capture local cross-channel interaction information, and multiplies each component of the weight sequence by the corresponding original data of each channel to obtain a feature map after the channel attention mechanism, which is used as the output feature of the feature extraction module.

7. According to claim 2, a motor imagery EEG signal classification system based on TimesNet and convolutional neural network is characterized in that the feature vector output by the fully connected classification module is processed by a softmax function to obtain scores belonging to each classification.

8. The motor imagery EEG signal classification system based on TimesNet and convolutional neural network according to claim 1 is characterized in that: in the model training and testing module, the EEG data of each subject is trained and tested separately.