[go: up one dir, main page]

Next Article in Journal
Free Vibration Analysis of Functionally Graded Shells Using an Edge-Based Smoothed Finite Element Method
Previous Article in Journal
A Scientific Decision Framework for Cloud Vendor Prioritization under Probabilistic Linguistic Term Set Context with Unknown/Partial Weight Information
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Multiple Transferable Recursive Feature Elimination Technique for Emotion Recognition Based on EEG Signals

1
School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093, China
2
Engineering Research Center of Optical Instrument and System, Ministry of Education, Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology, Shanghai 200093, China
*
Author to whom correspondence should be addressed.
Symmetry 2019, 11(5), 683; https://doi.org/10.3390/sym11050683
Submission received: 18 April 2019 / Revised: 7 May 2019 / Accepted: 15 May 2019 / Published: 17 May 2019
Figure 1
<p>V-A plane that defines four emotions with each subject’s self-assessment ratings and the subject-generic thresholds.</p> ">
Figure 2
<p>Illustration of influence of the alteration of the training set.</p> ">
Figure 3
<p>The schemes for building subject-specific training set for (<b>a</b>) TRFE and (<b>b</b>) M-TRFE.</p> ">
Figure 4
<p>Binary subject-specific classification accuracies on (<b>a</b>) arousal and (<b>b</b>) valence dimensions.</p> ">
Figure 5
<p>Influence of M-TRFE feature elimination on binary classification: (<b>a</b>) arousal accuracy, (<b>b</b>) valence accuracy, (<b>c</b>) arousal f1 score, (<b>d</b>) valence f1 score, with different amounts of features eliminated.</p> ">
Figure 6
<p>Binary classification accuracy on arousal (<b>a</b>) and valence (<b>b</b>) dimensions under the feature selection paradigms of subject-specific, RFE and S-TRFE and M-TRFE.</p> ">
Figure 7
<p>Classification performances of three strategies for four emotions, (<b>a</b>) joy, (<b>b</b>) peace, (<b>c</b>) anger and (<b>d</b>) depression using separate binary classifiers.</p> ">
Figure 8
<p>Illustration of M-TRFE feature transferring: (<b>a</b>) the number of features that were eliminated for each subject, (<b>b</b>) the best results of subject-specific RFE using OvO, and (<b>c</b>) OA of M-TRFE when the number of subjects employed increases.</p> ">
Versions Notes

Abstract

:
Feature selection plays a crucial role in analyzing huge-volume, high-dimensional EEG signals in human-centered automation systems. However, classical feature selection methods pay little attention to transferring cross-subject information for emotions. To perform cross-subject emotion recognition, a classifier able to utilize EEG data to train a general model suitable for different subjects is needed. However, existing methods are imprecise due to the fact that the effective feelings of individuals are personalized. In this work, the cross-subject emotion recognition model on both binary and multi affective states are developed based on the newly designed multiple transferable recursive feature elimination (M-TRFE). M-TRFE manages not only a stricter feature selection of all subjects to discover the most robust features but also a unique subject selection to decide the most trusted subjects for certain emotions. Via a least square support vector machine (LSSVM), the overall multi (joy, peace, anger and depression) classification accuracy of the proposed M-TRFE reaches 0.6513, outperforming all other methods used or referenced in this paper.

1. Introduction

Emotions are known as a group of intrinsic cognitive states of the human mind. It adds meanings to human activities and plays a vital role in human communication, intelligence, and perception [1]. An emotion can be triggered by a specific feeling and will eventually lead to a change in behavior [2]. Since emotions are closely associated with human activities and psychophysiological states, establishing intelligent emotion recognition is integral to achieve adaptive human-machine interaction (HCI). One preparatory work for emotion recognition is target emotion tagging, a process that assigns proper emotional labels to improve the efficiency of annotation methods of final classification performance [3]. Previous pieces of literature have proposed several emotion models. Some of them, such as Ekman’s and Parrot’s, are widely adopted but are poor in the term of the number of emotions (six emotions). The wheel of emotions by Plutchik and recently proposed a 3D model hourglass of emotions that is able to obtain complex emotions (more than 20 emotions in total). As the mainstream of DEAP-based studies, Russell’s valence-arousal (V-A) model is used in this literature [4,5,6,7]. The V-A plane with arousal score as a horizontal axis and valence score as vertical axis could be set up, from which each emotional state has an arousal dimension and a valence dimension [8]. Arousal scores range from inactivity to activity of one participant and the valence score measures the level of pleasure of him. Then, the V-A plane is ready to divide the target emotions (see Section 3.2).
Emotions can be expressed in both verbal and nonverbal manners. Therefore, it is important to build an HCI system that can recognize emotions by identifying facial or voice expressions of users [9]. The corresponding effective computing system must contain multifaceted processes. First, the HCI system should detect whether a specific emotion is expressed, and thus correctly label the emotional class (e.g., happiness or sadness) [10]. However, either facial or voice indicators are not always reliable. Past studies utilizing these indicators for HCI system emotion recognition show that subjects often intend to make their tones or manners in an exaggerated way to achieve a satisfactory performance [11]. Thus, emotion recognition via recording and analyzing physiological signals becomes a promising alternative [12]. Particularly, electroencephalography (EEG), with its non-invasive technique that easily yields input data for emotion classifiers, is becoming a preferred indicator [13]. EEG signals are immense in volume and high in dimension. For example, one single participant provides 8064-dimension original data in DEAP database, which are impossible to be handled directly. Another significant problem with emotion recognition via EEG is the response from each individual varies differently upon receiving the same affective stimuli. This is because emotions are personal and the evaluation should use an individual-specific assessment model. Since there is an existence of data distribution between subjects a long period of time is inevitable and required to train the classifiers. Furthermore, EEG signals could also be distributed differently in different days due to its non-stationarity. A proper model trained using the EEG data from a specific individual may not be well adapted for use on novel users, and therefore, feasible feature selection methods are imperative to transfer the useful information among individuals. Thus, the machine learning approach is adopted to extract useful information as clues for emotion recognition.
This paper focuses on the importance of selecting salient EEG indicators. All the algorithms mentioned below can be used for cloud services or non-cloud services. To examine high-dimensional EEG features, the recursive feature selection (RFE) combined with the least square support vector machine (LSSVM) was developed. The RFE-LSSVM has the capability to rank EEG features and selects the most relevant variables [14]. Choosing LSSVM over a vector machine (SVM) is because LSSVM shows less computational consumption [15]. Considering the need for cross-subject emotion recognition, it is reasonable to modify traditional RFE into the transferable recursive feature elimination (TRFE) [16]. This approach eliminates the EEG indicators that are not generic for all users and forms a set of robust EEG indicators that are steadily distributed among all training subjects and the specific novel testing subject.
With TRFE, the classifier does not necessarily require the corresponding specific training set up for the novel testing subject. By processing the reusable historical data collected from all other subjects, the training dataset is identified and produced. Following this concept, the development of TRFE, single TRFE (S-TRFE) and multiple TRFE (M-TRFE) are proposed. Both algorithms are based on a novel transferring set that contains the most trustful features from other subjects. While the S-TRFE algorithm directly adds the transferring set to the entire training set of one subject, M-TRFE removes some worst features from the entire training set of one subject and replaces them with the given transferring set to improve classification performance. In addition, M-TRFE also tries to select the most trusted subjects with better performance in cross-subject emotion recognition. Thus, the more a subject is trusted, the more he is donating to the transferring set. This process can be described as getting rid of the outliers who have not reacted commonly as most people do.
Based on this M-TRFE algorithm, the expected cross-subject classifier should have better performance. The accuracies given by this classifier are expected to be higher than others. Throughout this entire process, we exploit the DEAP database as the working resource.
To be concise, in the rest of the paper, TRFE will be used as a collective name that encompasses a series of RFE based cross-subject schemes. The original TRFE algorithm will be renamed as general TRFE (G-TRFE) to make a distinction. The newly proposed M-TRFE algorithm will be compared against all strategies previously mentioned, as well as the subject-specific (SS) case on both binary and multiclass emotion recognition.
The rest of the paper consists of several sections. Section 2 dedicates to the summary of related works that inspire this work, and Section 3 provides a short description of DEAP dataset, the EEG preprocessing scheme, and the feature extraction methods on DEAP. Current section will also demonstrate the workflow of LSSVM and the detailed process of M-TRFE as well. Section 4 consists of binary and multiclass emotion recognition, where different cross-subject or subject-specific methods are expounded, tried, and compared. The last two sections focus on result analysis, main contribution, the implication of this work, its limitations and its potentials.

2. Related Works

Emotion recognition is utilized in many fields. He et al. proposed a convolutional neural network (CNN) that recognizes emotion from an image by combining a binary classification network and a deep network [17]. In addition, a facial recognition system has been applied to evaluate the quality of distance teachings [18]. In speech analysis, emotion recognition is implemented by using the extreme learning machine (ELM) [19]. Music, in which emotions are expressed, can be analyzed to tell the difference between contemporary commercial music and classical singing techniques [20]. Classification performances of speech and music recognition systems are not ideal (around 50%), but facial and voice recognition systems have achieved high accuracies of 0.8170 and 0.8753, respectively.
Aside from facial and vocal features, physiological features have also been widely used in emotion recognition. To be more specific, EEG signals are also investigated via machine learning based classifiers. For instance, gender recognition with entropy measurement is achieved in Hu et al. [21]. The connection between mental fatigue and aging has been studied. The recognition of mental fatigue was found to be efficient when adaptive weights switch in deep belief networks (DBN) [22,23]. Even though DBN has also been applied in the recognition of emotion more studies use SVM which combines feature smoothing or selection methods, such as canonical correlation analysis (CCA) and principal component analysis (PCA) [24,25,26]. An end-to-end model based on CNN is used to reduce the cost of designing the feature set, and as a result, the average accuracy of 0.7548 was reported [27]. In a recent study, Tong et al. [28] combined the International Affective Picture System (IAPS) that sorted eight valence levels with similar arousal values, with nonlinear feature based SVM. These EEG-based emotion recognition methods are very encouraging and have gained widespread attention.
Several studies have already been done and shown the efficacy of various feature selection methods and their use in EEG based emotion recognition. Zhang et al. combined feature extraction methods of empirical mode decomposition and sample entropy [29]. Atkinson and Campos’ work integrated mutual information based on EEG feature selection and kernel classifiers [30]. A novel feature termed as DE-PCCM proposed by Li et al. has good outcomes when a differential entropy (DE) feature extraction was employed [31].
RFE approaches are of particular interest to us due to the previous work which demonstrates the applicability of these approaches to emotion recognition. SVM-RFE detecting scalp spectral dynamics of interest (SSDOIs) has promising clinical applications [32]. Another modification of the RFE approach, D-RFE (stands for dynamical-RFE), was proposed to improve the inter-class discrimination [33]. In a series of previous works, we investigated OFS classification using LSSVM based on RFE [34]. Motivated by these studies, we adopted the supervised learning methodology, the 2D V-A plane to target four emotions and M-TRFE algorithm for EEG feature selection.

3. Methods

3.1. EEG Datasets for Effective Modeling

In this study, the DEAP (a database for emotion analysis using physiological signals) database was used to validate the proposed machine learning-based feature selector. Total of 32 subjects with (average age was 26.9, 50% of them were female) took part in the experiment for physiological data acquisition. The International 10–20 System was implemented for recording EEG signals in which 32-channel data were collected under a sampling rate of 256 Hz [35]. There were 40 video clips (i.e., 40 trials, each trial lasted about 1 min) prepared for each participant as the emotional stimuli, which equates to 40 trials for each subject to complete. The physiological responses were simultaneously recorded while the participant was watching the video. All subjects accomplished a self-assessment after each trial where arousal, valence, dominance, liking, and familiarity scales were labeled. With the exception of the familiarity scale (range 1–5), the remaining four rating scales ranged from 1 to 9. Then the VA-model was used to determine the target emotion classes [36].

3.2. Feature Extraction and the Target Emotion Classes

The Butterworth filter with a cutoff frequency of 4.0 and 45.0 Hz was used to filter the noise in the EEG data [37]. Then, an independent component analysis (ICA) was employed to eliminate muscular artifacts [38]. In each trial, 60-s continuous EEG signals were selected and split into three segments: 3-s baseline segment, 6-s (10%) validating segment, and 54-s working segment. The validating segment was used to rank features and avoid overfitting in the M-TRFE model, while the working segment was used to select features and perform the classifier training and testing. The baseline segment was discarded in this work because it is collected before the subject watches the video.
In this work, 11 channels out of 32 channels were picked. These channels are F3, F4, Fz, C3, C4, Cz, P3, P4, Pz, O1 and O2. This particular choice of channels follows the channel employment in previous work of Zhang et al. [39]. Overall, 137-dimensions EEG features were extracted, which consists of 60 frequency domain features and 77 time domain features. By using a fast Fourier transformation, the frequency features (60 power features, 16 power difference features) were prepared. In each channel, the power features were computed on four frequency bands, i.e., theta (4–8 Hz), alpha (8–12 Hz), beta (12–30 Hz) and gamma (30–45 Hz). Power difference features were employed to detect the variation in cerebral activity between the left and right cortical areas. There are four channel pairs, F4-F3, C4-C3, P4-P3 and O2-O1, used for power differences extraction with each pair, contributing four features of four bands. For each channel, seven temporal features were computed as seven indexes: mean, variance, zero crossing rate, Shannon entropy, spectral entropy, kurtosis, and skewness. All features were standardized with mean = 0 and s.d. = 1. The detailed descriptions of the features are shown in Table 1.
Emotion classification achieved based on supervised learning requires predetermined emotion labels. In the DEAP database, participants used self-assessment manikins to rate the valence and arousal levels in the range from 1 to 9. The subject rated valence or arousal levels from the lowest of 1 to the highest of 9. A threshold is conventionally set up and calculated to determine high/low valence or arousal classes. The value of the threshold point here was determined in a participant generic manner. With each of 32 subjects selected rating values, the mean values of both valence and arousal indexes from all subjects and trials were calculated. For every subject’s 40 arousal ratings a1, a2, …, a40 (aiR2), arousal threshold point c1 is computed as follows:
c = j = 1 32 i = 1 40 a i 40 · 32 , ( j = 1 , 2 , 32 )
The same process using valence ratings was used to compute the valence threshold point, c2. It was found that c1 = 5.2543 and c2 = 5.1567 were the threshold values for arousal and valence dimensions. The ratings above c1 were assigned as the state of high arousal and the ratings above c2 were the state of high valence.
The entire V-A plane was split into 4 parts: HVHA (high valence high arousal), HVLA (high valence low arousal), LVHA (low valence high arousal), and LVLA (low valence low arousal). This is illustrated in Figure 1. Finally, the four emotions of joy, peace, depression and anger were assigned to each respective quadrant in the V-A plane.

3.3. Multiple Transferable Feature Elimination Based on LSSVM

M-TRFE was developed via LSSVM due to its merits in faster-training and better performance in avoiding overfitting. Here is the principle to select the feature instance. Given the training set D = { ( x i ,   y i )   |   i = 1 , 2 , , l } with the input data x i R n and the corresponding output labels y i { + 1 , 1 } . The nonlinear mapping φ ( x ) was used to generate a higher dimensional feature space aiming at finding the optimal decision function,
y ( x ) = w · φ ( x ) + b
In Equation (2), w stands for the weight vector of the classification separating hyperplane and y ( x ) is the linear estimation function in feature space. To achieve minimization of structural risk, the scheme was carried out as below:
min J ( w ,   ζ ) = 1 2 | | w | | 2 + γ i = 1 l ζ i 2 s . t .   y i = φ ( x i ) · w + b + ζ i
where γ is the regularization parameter for adjusting the punishment degree of training error, | | w | | 2 is in control of the complexity of the model, and the last term i = 1 l ζ i 2 is an empirical error on the training set, where the slack variable ζ i is introduced in case of nonlinear separable of the instances in two classes. The Lagrangian function can be constructed with the kernel function K ( x i , x j )   = φ ( x i ) · φ ( x j ) to find solutions of a linear equation system. Applying the least square method, a nonlinear prediction model is exposed via kernel function K :
y ˜ = α i K ( x , x i ) + b
According to the equations above, M-TRFE measures if a feature is salient by checking the classification margin and the loss of the margin when the kth feature is eliminated, i.e.,
Δ Φ = | | | w | | 2 | | w ( k ) | | 2 |
In Equation (5), w ( k ) is the weight vector of the classification plane with the kth feature eliminated. If the elimination of a particular feature leads to the largest Δ Φ , the corresponding feature is considered as the most influential one.
The goal of M-TRFE is to determine a set of best indicators among a group of participants. It is noted that the binary LSSVM is not capable of the four-class classification task. Here the one against one ensemble (OvO) of multiclass classifiers is utilized to fulfill the task. With OvO structure, each two emotion classes are tackled as a pair via an M-TRFE-LSSVM model. The details are shown as follows.
Given a sample set D = { ( x i ,   v i )   |   i = 1 , 2 , , N } with v i { 1 , 2 , , l } , initialize the feature set S = { 1 , 2 , , D } , the feature-ranking set R = [ ] , and the feature ranking vector p = [ ] . Combine two training samples as a pair, and eventually generate l ( l 1 ) / 2 novel training sample, the last classifier can be built as:
x j = { ( x i , y i ) | i = 1 , 2 , N l 1 + N l } ; j = l ( l 1 ) 2 1 , , l ( l 1 ) 2 ; when   v i = l 1 , y i = 1 ; when   v i = l , y i = 1 .
In the first step, we can use the obtained x j to train an LSSVM model with the computed weight vector w j ( j = 1 , 2 , , l ) . Then, the sorting criterion score can be calculated as follows,
p = arg min k j w j k 2 , k = 1 , 2 , , S .
Update the feature ranking set as R = [ p , R ] and delete the feature in S. This process repeats till S = [ ] .
In M-TRFE, credible training data contain the best feature instances from others. At the same time, the selected training data eliminate some of the worst performing features from the original training set. M-TRFE will also rate the pick of only a few subjects to take part in the building of this set. The influence of the variation of the training set of the TRFE concept is illustrated in Figure 2. The construction of the M-TRFE novel training set is unfolded in Figure 3a.
Notably for the use of multiclass M-TRFE, for the training set D = { ( x i ,   y i )   |   i = 1 , 2 , , n } with the multiclass label y i { 1 , 2 , 3 , 4 } , several separate binary classifiers robustly analyze each emotion and encode the label into binary values y i { 1 , + 1 } . This would give each emotion a feature ranking and subject selection. To fulfill the multiclass subject-generic emotion feature selection, a mutual feature-ranking list is generated based on four separate rankings of joy, peace, anger and depression, to detect the best features and most trusted subjects for each emotion. We consider one subject is more trusted if he achieves better results in cross-subject classification accuracy. The more one subject is trusted, the more he contributes to the transferring training set. Meanwhile, the least trusted ones stop contributing the set as is shown in Figure 3b. The weighted score of each feature can be averaged from all ranking lists.
W mutual = W joy + W peace + W anger + W depression when   W = 1 0.1 r ,   r = 0 , 1 , 2 , , 9
In Equation (8), if one feature is determined as the worst feature, the ranking index r-value will be 0. The r-value of the second worst feature would be considered as 1 and so on. By choosing the least trusted 10 features, rankings for different emotions would be given. In fact, the W value evaluates how favorable one feature is. Moreover, instead of using separate feature ranking arrays, the mutual array can be calculated in the proposed M-TRFE multiclass model.
The workflow of M-TRFE is as follows. For a given high emotion state (V-A or multiclass) as VH and the low emotion state as VL, the corresponding state centers v H and v L are computed. For the j th feature of the i th subject, we define the Euclidean distance between the original EEG feature set to a novel transferring set as
ϵ = | | x j v H i | | .
The H-value can be calculated as
H = N O i ϵ j = 1 N O i ϵ j = 1 N O i ( N O i ϵ j = 1 N O i ϵ )
In Equation (10), N O i is the cardinal number of Oi, while Oi is the newly extended space. H < 0 indicates the feature value is far away from VH and such feature will be eliminated from the high class. The details of M-TRFE are written in the form of pseudo codes and are given in Table 2 and Table 3.
There are several details that need to be explained in Table 2. s in line 2 stands for the number of subjects that took part in the trials, while f value in line 4 controls the number of folds if L-fold cross-validation technique is applied. In this study, s = 32 and f = 10. j1 from line 9 starts the subject ranking. A i records cross-subject performance of subject i.
In Table 3, the high state of emotion is taken as an example. The low state of emotion can also track the pseudo codes. There are several parameters that need to be explained. L = 137 in line 10 represents the dimensionality of the feature set. Since L is a prime number in this work, the step length of iteration of each elimination has to be taken as 1. In line 15, dH (and dL) quantifies the distance between the original set and transferring set for the high (and low) class, and the distance difference D ˜ ( r 1 ) is considered equally influential as LSSVM margin loss w ˜ ( r 1 ) by taking λ 1 = λ 2 = 0.5 . An auxiliary function f a has also been used in the pseudo codes, which is introduced to simplify the representation of the algorithm:
f a [ w ( k ) ] = L · w ( k ) k = 1 L w ( k ) L · k = 1 L ( w ( k ) 1 L · k = 1 L w ( k ) ) 2
To evaluate the classification performance of the proposed feature selection model, several assessment metrics are introduced as accuracy, F1 score, and kappa value.

4. Results

All the experiments and following results were carried out via Matlab R2016b, with the computer running on Windows 10 operating system with Intel® CoreTM i5-7200U CPU @ 2.50 GHz 2.71 GHz and 8 GB RAM.

4.1. Data Split and Cross-Validation Technique

In this subsection, several strategies for data splits based on different cross-validation techniques were tested. Since our feature extraction had enlarged the feature space, we needed to take different proportions of data randomly (not orderly) from the working segment as training sets. As the training/test set was completely different between each repetition, the subject-specific emotion recognition was run for five times to test the stability of the random use of the working segment. The value of accuracy is shown in Figure 4, the results of which correspond to random data splits under hold-out or 10-fold cross validation conditions.
According to Figure 4, the hold-out cross-validation only yields accuracy values around 50%. Therefore, it would be impractical to evaluate generalization capability. To tackle this issue, we used the 10-fold cross-validation to achieve an acceptable classification performance (AVG arousal accuracy = 0.6549, AVG valence accuracy = 0.6865). In contrast to hold-out cross-validation, 10-fold cross validation divides the validating segment equally into ten small folds, with each fold estimating the accuracy. This was found to be a better model to enhance the classification performance. Thus, the analysis in the rest of the paper all employ 10-fold cross-validation. Figure 4 also shows that the classification results across different repetitions do not significantly fluctuate, which indicate the random use of the working segment is feasible.

4.2. Cross-Subject Feature Selection and Binary Classification

There are several strategies to realize binary cross-subject emotion recognition. In the strategy of S-TRFE, the entire training set from one subject remains, while different amounts of relevant features from other subjects are added to create a novel training set. We gradually increased the use of the transferring set, but despite alterations to the set, we found that the classification performance was generally unchanged. The value of F1-score did not significantly vary as well. On the dimension of the valence, accuracy ranged from 0.6865 to 0.6875. However, for the arousal dimension, the accuracy dropped from 0.6549 to 0.6470.
In the case of M-TRFE, two key factors of the paradigm are given in Table 3 and Table 4. We labeled the participant who reached the highest classification accuracy in the direct cross-subject scheme as the most trusted subject, and the assigned ranks are listed in Table 4. In this direct scheme, the RFE based feature selection was not performed. Instead, all subjects were directly involved in training the classifier. The AVG arousal and valance accuracy reached 0.5089 and 0.4961. The unsatisfactory performance of direct cross-subject scheme paradigm confirmed the notion that cross-subject emotion recognition faces tough obstacles.
The worst features given by M-TRFE feature ranking were ranked and are presented in Table 5 with their corresponding physiological significance. The PSD features from the beta band were unwelcomed in the binary classification.
For all 32 subjects, the mean arousal and valence accuracies peaked when the worst feature was eliminated. As a result, the highest value of 0.6531 for arousal and 0.6867 for valence dimension were determined. Figure 5 reveals the variation of the classification performance when different numbers of features were excluded. With the worst feature eliminated, the classification performance achieved the highest value. Although all these metrics were slightly improve compared to SS, the enhancement was not significant.
M-TRFE paradigm uses a certain amount of relevant features from other most trusted subjects to replace an equal amount of the least relevant features of one specific subject that was eliminated. With the single most trusted subject contributing to the transferring set and with the most relevant feature of this subject being employed, we found that the classification accuracies on binary emotional dimensions peak. The classification performance on SS, RFE, S-TRFE and M-TRFE for each subject is illustrated in Figure 6. M-TRFE overtakes other cross-subject methods, yet inferior to SS. However, with all cross-subject methods analyzed, the recognition performances were actually close.
The average accuracies of all 32 participants are listed in Table 6. With regards to S-TRFE, altering the number of transferring features dis not influence the classification performance, and all the indexes remained the same except that the F1 score of arousal decreases progressively with the number of feature increases. Thus, S-TRFE merely shows the impact of transferring features, and the specific subject himself leads the classification performance. M-TRFE, meanwhile, performs the best when the most trusted subject was donated and its performance surpassed S-TRFE. We also used the G-TRFE algorithm, which was proposed by Yin in 2017 and was the inspiration of this work. However, it seemed not suitable for our feature extraction. The arousal accuracy was found to be 0.5580, and the valence accuracy was 0.5860. Unfortunately, accuracies of both dimensions were worse than M-TRFE.

4.3. Multiclass Cross-Subject Emotion Recognition

As is illustrated in Figure 1, a total of four states of emotions were extracted. Especially, by converting the multiclass task into several binary classifications, we took four separate binary classifiers to identify the worst feature for each emotion state and the most trusted subject for the transferring feature set. Therefore, each emotion would have its most trusted subject. The least credible features and most trusted subjects were ranked and used to implement M-TRFE, and these features are presented separately in Table 7 and Table 8. According to Equation (8), the over-all most trusted subjects and most credible features could be calculated and titled as “mutual”, aiming to identify the worst features and most trusted subjects in multi-classification for OvO structures. Additionally, the corresponding physiological significances of this mutual ranking were given. We applied several previous methods, including S-TRFE, M-TRFE, and G-TRFE in this subsection.
With the M-TRFE implemented, classification accuracies given by separate classifiers were all above 0.7 and the overall accuracy (OA) reaches 0.7538. Peace was considered as the best performing emotion, which gives an OA of 0.8932 and F1 score of 0.8025. Notably, for subject 3, the classification accuracy of class joy achieved 100%. The perfect recognition of the emotion anger was found in subjects 23 and 26.
The results of three different strategies are illustrated in Figure 7. G-TRFE was the worst performer (OA = 0.5390). S-TRFE performed better (OA = 0.6811) but it was still inferior to M-TRFE (OA = 0.7538) for all emotions. However, it should be noted that the achieved results are specific for only one particular emotion and are not the actual results of multiclass classification.
It should be mentioned that only one worst feature was removed in the binary state case, according to M-TRFE. During multiclass classification using OvO structure, more feature instances and more trusted subjects were available for M-TRFE to develop. As previously stated, the transferring set for OvO structure was decided by the mutual rankings given by separate binary classifiers. The results of feature elimination are depicted in Figure 8a,b and the corresponding eliminated amount of features are also shown. Since the maximum number of the eliminated features was limited to eighteen, the maximum number of other subjects employed would also be limited to eighteen. Under that condition, every subject employed provided one’s most relevant feature to the transferring set. Notably, when the two most trusted subjects were donated to the transferring set, M-TRFE performed the best, which can be seen in Figure 8c.
The results of M-TRFE, SS, S-TRFE and G-TRFE are all listed in Table 9. With the highest values of performance indexes including the kappa value and OA, M-TRFE was still the best choice for cross-subject emotion recognition. The kappa value suggests that there is a moderate level of agreement between the actual class and the predicted class of M-TRFE, which is similar to SS. The p-value of the one-way ANOVA between cross-subject schemes and SS was calculated and revealed that significant variation did exist. Meanwhile, the other two strategies were only deemed as a fair agreement. M-TRFE also had a balanced performance on all indexes. It is reasonable to conclude that M-TRFE is feasible and excellent on multi-classification using OvO structure.

5. Discussion

Due to the uncertainty to recognize human emotion through EEG, there is currently no sufficient knowledge that can be used to find the optimal machine learning method for feature selection. In this paper, a classical LSSVM-based feature selection algorithm was formed to resolve the existing cross-subject emotion classification problem. A novel EEG feature set was extracted from DEAP database to meet the cross-subject need. For physiological signals like EEG, different participants can have distinct reactions to the same stimuli. In this study, TRFE itself is described as an ideology of transferring historical data. Several other cross-subject algorithms based on TRFE and algorithms independent of TRFE are tested in comparisons. These algorithms all demand a delicate balance between an individual and the other individuals when compiling a novel training set. The proposed M-TRFE was exactly designed to offset this individual variation. It introduces the transfer learning principle that retains the shared information for a group of individuals. In other words, M-TRFE emphasizes the common ground in human emotion.
The feature extraction of this work is unique. The labels of DEAP database are rated for the entire duration of the video clips but we extended the labels every two seconds. Since the subjects are all informed of the video contents before the trials, it is reasonable to believe the emotion remains consistent through the entire course. The expansion of the feature set includes more emotion samples to the experiments and has a stronger influence on the results.
Through many experiments including binary classification and multiclass classification, M-TRFE is a particularly outstanding cross-subject method. Unlike other algorithms based on RFE, M-TRFE requires a careful selection of trusted subjects who are allowed to contribute to the novel training set. In order to identify these subjects, some special steps were designed to take. In binary classification, we search for those who perform better in the direct scheme. In the multiclass case, four separate binary classifiers directly reduce the number of labels from four to two and then forms a mutual ranking. Accordingly, the PSD of the beta band appears to be the least contributive physiological significance in multiclass classification. The binary classifiers also reached this conclusion. Meanwhile, OvO is a classical classifier ensemble that prevents overfitting and has given out the final results of multiclass classification.
On binary affective states, the allocation of the training/testing set and of 10-fold cross-validation was determined. Both were used throughout the experiments. Compared to subject-specific results, all the cross-subject schemes seem to be stagnant in their development. This is mostly because the feature ranking and elimination only permit the single worst feature to be eliminated, which leaves no room for M-TRFE to develop. On the other hand, multiclass classification gives the transferring training set a sufficient fusion. However, the number of trusted subjects that M-TRFE chooses is still limited. M-TRFE attains its best performance when only two subjects are involved. This might be partly due to the divergence of human emotion would be enlarged when affective computing recruits more individuals to participate in the cross-subject task. Even so, M-TRFE still becomes the preferred paradigm of most of the indexes, and even exceeds SS using OvO structure.
Furthermore, M-TRFE had not only better classification accuracy but also a faster running speed. The running period of MTRFE-LSSVM was significantly shorter by 86.97% than GTRFE-LSSVM. The latter appears to be inordinately time-consuming due to the oceanic design of feature set, costing 4784.80 s per training period verse 623.22 s by MTRFE-LSSVM. Moreover, M-TRFE efficiently reduces resources waste by selecting the best features from other individuals and putting them into use. This resource efficiency is exactly what the principle of the transfer learning expects. On the other hand, since the training set of S-TRFE is built by raising the dimension, its testing set dimension will also be raised. This occurrence is actually contrary to the concept of RFE. However, M-TRFE modifies this flaw by maintaining the dimension and reinforcing the performance. Thus, all the results bring to the conclusion that M-TRFE is far more superior in cross-subject emotion recognition.
Our work is also compared to other recent studies on DEAP dataset in Table 10. Our model achieves top performances in classification performance categories in the demonstration. The OA of binary classification was calculated as the average accuracy of the arousal and valance dimension. Here are some necessary explanations to the abbreviations in the table: SVD = singular value decomposition, mRMR = minimum redundancy maximum relevance, EC = evolutionary computation, FAWT = flexible analytic wavelet transform.

6. Conclusions

It is an intractable and challenging task for cross-subject emotion recognition to distinguish human individual emotion. However, the generality of human being and their emotions guarantees the potential for automated systems to perform cross-subject recognition. In this paper, the cross-subject emotion recognition has been carried out with the novel M-TRFE feature selection method on both binary and multiclass classification problems. M-TRFE manages not only the selection of feature instances but also the selection of individuals. By choosing participants who react closer to the common reaction, M-TRFE performs similar to the subject-specific recognition on the binary affective state and it prevails over all methods on multiclass classification. Throughout the work, LSSVM was applied to accomplish the selection. The binary classification rate achieved 0.6494 and 0.6898 on arousal and valance dimensions. In the case of multiclass, OA achieved 0.6513. These results have outperformed all other methods applied in this paper and most of the recently reported studies on the DEAP database. In general, M-TRFE has made cross-subject emotion recognition more efficient and precise with less resource waste.
In future work, we will look for more stable and more accurate classifiers to improve emotion recognition. We will also combine some other physiological signals such as eye gaze, GSR, blood pressure, etc., with the EEG signals. In addition, compound emotions such as anxiety can be a new direction for further research, under V-A or other emotion models. Although M-TRFE has been proven to be an excellent solution to cross-subject emotion classification, there are still flaws that need further investigation and solution. More complex multiclass tasks with larger quantities of emotions and participants will challenge the performance of M-TRFE. If there are too many subjects involved, detecting the most trusted subjects will be difficult. The improvement of M-TRFE will also be a topic for future research.

Author Contributions

Investigation, formal analysis, visualization, writing—original draft preparation, J.C.; conceptualization, methodology, J.C. and Z.Y.; writing—review and editing, Z.Y.; supervision, validation, W.C.

Funding

This research was funded by the National Natural Science Foundation of China under Grant No. 61703277 and the Shanghai Sailing Program under Grant No. 17YF1427000.

Acknowledgments

The authors would like to give our sincere gratitude to Wenwei Yan and Jielin Yan, who have provided a lot of help to polish this article.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Panksepp, J. Affective Neuroscience: The Foundations of Human and Animal Emotions; Oxford University Press, Oxford: New York, NY, USA, 2005. [Google Scholar]
  2. Schacter, D.L.; Gilbert, D.T.; Wenger, D.M.; Nock, M.K. Psychology, 3rd ed.; Worth: New York, NY, USA, 2014. [Google Scholar]
  3. Siegert, I.; Böck, R.; Vlasenko, B.; Philippou-Hübner, D.; Wendemuth, A. Appropriate emotional labelling of non-acted speech using basic emotions, Geneva emotion wheel and self-assessment manikins. In Proceedings of the 2011 IEEE International Conference on Multimedia and Expo, Barcelona, Spain, 11–15 July 2011; pp. 1–6. [Google Scholar]
  4. Bradley, M.M.; Lang, P.J. Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 1994, 25, 49–59. [Google Scholar] [CrossRef]
  5. Parrott, W.G. Emotions in Social Psychology: Essential Readings; Psychology Press: Philadelphia, PA, USA, 2001. [Google Scholar]
  6. Ekman, P.; Dalgleish, T.; Power, M. Handbook of Cognition and Emotion; Wiley: Chichester, UK, 1999. [Google Scholar]
  7. Cambria, E.; Livingstone, A.; Hussain, A. The Hourglass of Emotions. Cogn. Behav. Syst. 2012, 7403, 144–157. [Google Scholar]
  8. Mehrabian, A. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Curr. Psychol. 1996, 14, 261–292. [Google Scholar] [CrossRef]
  9. Keltner, D.; Ekman, P. Facial Expression of Emotion, 2nd ed.; Guilford Publications: New York, NY, USA, 2000. [Google Scholar]
  10. Zhang, Q.; Chen, X.; Zhan, Q.; Yang, T.; Xia, S. Respiration-based emotion recognition with deep learning. Comput. Ind. 2017, 92–93, 84–90. [Google Scholar] [CrossRef]
  11. Tan, D.; Nijholt, A. Human-Computer Interaction Series; Springer: London, UK, 2010. [Google Scholar]
  12. Meehan, K.B.; Panfilis, C.D.; Cain, N.M.; Antonucci, C.; Soliani, A.; Clarkin, J.F.; Sambataro, F. Facial emotion recognition and borderline personality pathology. Psychiatry Res. 2017, 255, 347–354. [Google Scholar] [CrossRef]
  13. Christensen, J.; Estepp, J.; Wilson, G.; Russell, C. The effects of day-to-day variability of physiological data on operator functional state classification. Neuroimage 2012, 59, 57–63. [Google Scholar] [CrossRef]
  14. Yin, Z.; Fei, Z.; Yang, C.; Chen, A. A novel SVM-RFE based biomedical data processing approach: Basic and beyond. In Proceedings of the IECON 2016—42nd Annual Conference of the IEEE Industrial Electronics Society, Firenze, Italy, 24–27 October 2016; pp. 7143–7148. [Google Scholar]
  15. Shao, Z.; Yang, S.L.; Gao, F.; Zhou, K.L.; Lin, P. A new electricity price prediction strategy using mutual information-based SVM-RFE classification. Renew. Sustain. Energy Rev. 2017, 70, 330–341. [Google Scholar] [CrossRef]
  16. Yin, Z.; Wang, Y.X.; Liu, L.; Zhang, W.; Zhang, J.H. Cross-Subject EEG Feature Selection for Emotion Recognition Using Transfer Recursive Feature Elimination. Front. Neurorobot. 2017, 11, 1662–5218. [Google Scholar] [CrossRef] [PubMed]
  17. He, X.; Zhang, W. Emotion recognition by assisted learning with convolutional neural networks. Neurocomputing 2018, 291, 187–194. [Google Scholar] [CrossRef]
  18. Yang, D.; Alsadoon, A.; Prasad, P.W.C.; Singh, A.K.; Elchouemi, A. An emotion recognition model based on facial recognition in virtual learning environment. Procedia Comput. Sci. 2018, 125, 2–10. [Google Scholar] [CrossRef]
  19. Kaya, H.; Karpov, A.A. Efficient and effective strategies for cross-corpus acoustic emotion recognition. Neurocomputing 2018, 275, 1028–1034. [Google Scholar] [CrossRef]
  20. Hakanpää, T.; Waaramaa, T.; Laukkanen, A.M. Emotion recognition from singing voices using contemporary commercial music and classical styles. J. Voice 2018. [Google Scholar] [CrossRef]
  21. Hu, J. An approach to EEG-based gender recognition using entropy measurement methods. Knowl. Based Syst. 2018, 140, 134–141. [Google Scholar] [CrossRef]
  22. Arnau, S.; Möckel, T.; Rinkenauer, G.; Wascher, E. The interconnection of mental fatigue and aging: An EEG study. Int. J. Psychophysiol. 2017, 117, 17–25. [Google Scholar] [CrossRef]
  23. Yin, Z.Y.; Zhang, J. Cross-subject recognition of operator functional states via EEG and switching deep belief networks with adaptive weights. Neurocomputing 2017, 260, 349–366. [Google Scholar] [CrossRef]
  24. Li, X.; Zhang, P.; Song, D.; Yu, G.; Hou, Y.; Hu, B. EEG based emotion identification using unsupervised deep feature learning. In Proceedings of the SIGIR2015 Workshop on Neuro-Physiological Methods in IR Research, Santiago, Chile, 9–13 August 2015. [Google Scholar]
  25. Chen, S.; Gao, Z.; Wang, S. Emotion recognition from peripheral physiological signals enhanced by EEG. In Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China, 20–25 March 2016; pp. 2827–2831. [Google Scholar]
  26. Shahnaz, C.; Shoaib-Bin-Masud; Hasan, S.M.S. Emotion recognition based on wavelet analysis of Empirical Mode Decomposed EEG signals responsive to music videos. In Proceedings of the 2016 IEEE Region 10 Conference (TENCON), Singapore, 22–25 November 2016; pp. 424–427. [Google Scholar]
  27. Wen, Z.; Xu, R.; Du, J. A novel convolutional neural networks for emotion recognition based on EEG signal. In Proceedings of the 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC), Shenzhen, China, 15–18 December 2017; pp. 672–677. [Google Scholar]
  28. Tong, J.; Liu, S.; Ke, Y.F.; Gu, B.; He, F.; Wan, B.; Ming, D. EEG-based emotion recognition using nonlinear feature. In Proceedings of the 2017 IEEE 8th International Conference on Awareness Science and Technology (iCAST), Taichung, China, 8–10 November 2017; pp. 55–59. [Google Scholar]
  29. Zhang, Y.; Ji, X.; Zhang, S. An approach to EEG-based emotion recognition using combined feature extraction method. Neurosci. Lett. 2016, 633, 152–157. [Google Scholar] [CrossRef]
  30. Atkinson, J.; Campos, D. Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Syst. Appl. 2016, 47, 35–41. [Google Scholar] [CrossRef]
  31. Li, H.; Qing, C.; Xu, X.; Zhang, T. A novel DE-PCCM feature for EEG-based emotion recognition. In Proceedings of the 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC), Shenzhen, China, 15–18 December 2017; pp. 389–393. [Google Scholar]
  32. Oostenveld, R.; Praamstra, P. The five percent electrode system for high-resolution EEG and ERP measurements. Clin. Neurophysiol. 2001, 112, 713–719. [Google Scholar] [CrossRef]
  33. Hidalgo-Muñoz, A.R.; López, M.M.; Santos, I.M.; Pereira, A.T.; Vázquez-Marrufo, M.; Galvao-Carmona, A.; Tomé, A.M. Application of SVM-RFE on EEG signals for detecting the most relevant scalp regions linked to affective valence processing. Expert Syst. Appl. 2013, 40, 2102–2108. [Google Scholar] [CrossRef]
  34. Yin, Z.; Zhang, J. Operator functional state classification using least-square support vector machine based recursive feature elimination technique. Comput. Methods Prog. Biomed. 2014, 113, 101–115. [Google Scholar] [CrossRef]
  35. Hamada, Y.; Elbarougy, R.; Akagi, M. A method for emotional speech synthesis based on the position of emotional state in Valence-Activation space. In Proceedings of the Signal and Information Processing Association Annual Summit and Conference (APSIPA), Siem Reap, Cambodia, 9–12 December 2014; pp. 1–7. [Google Scholar]
  36. Koelstra, S.; Mühl, C.; Soleymani, M.; Lee, J.S.; Yazdani, A.; Ebrahimi, T.; Pun, T.; Nijholt, A.; Patras, I. DEAP: A database for emotion analysis using physiological signals. IEEE Trans. Affect. Comput. 2012, 3, 18–31. [Google Scholar] [CrossRef]
  37. Atasoyu, M.; Metin, B.; Kuntman, H.; Cicekoglu, O. Simple realization of a third order Butterworth filter with MOS-only technique. AEU 2017, 81, 205–208. [Google Scholar] [CrossRef]
  38. Chen, X.; Liu, A.; Chen, Q.; Liu, Y.; Zou, L.; McKeown, M.J. Simultaneous ocular and muscle artifact removal from EEG data by exploiting diverse statistics. Comput. Biol. Med. 2017, 88, 1–10. [Google Scholar] [CrossRef]
  39. Zhang, J.; Yin, Z.; Wang, R. Recognition of mental workload levels under complex human-machine collaboration by using physiological features and adaptive support vector machines. Hum. Mach. Syst. 2015, 45, 200–214. [Google Scholar] [CrossRef]
  40. Naser, D.S.; Saha, G. Recognition of emotions induced by music videos using DT-CWPT. In Proceedings of the Indian Conference on Medical Informatics and Telemedicine (ICMIT), Kharagpur, India, 28–30 March 2013; pp. 53–57. [Google Scholar]
  41. Zhu, Y.; Wang, S.; Ji, Q. Emotion recognition from users’ EEG signals with the help of stimulus videos. In Proceedings of the 2014 IEEE international conference on multimedia and expo (ICME), Chengdu, China, 8–12 July 2014; pp. 1–6. [Google Scholar]
  42. Feradov, F.; Ganchev, T. Detection of negative emotional states from electroencephalographic (EEG) signals. Annu. J. Electron. 2014, 8, 66–69. [Google Scholar]
  43. Candra, H.; Yuwono, M.; Handojoseno, A.; Chai, R.; Su, S.; Nguyen, H.T. Recognizing emotions from EEG subbands using wavelet analysis. In Proceedings of the 2015 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC), Milan, Italy, 25–29 August 2015; pp. 6030–6033. [Google Scholar]
  44. Nakisa, B.; Rastgoo, M.N.; Tjondronegoro, D.; Chandran, V. Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors. Expert Syst. Appl. 2018, 93, 143–155. [Google Scholar] [CrossRef]
  45. Gupta, V.; Chopda, M.D.; Pachori, R.B. Cross-Subject Emotion Recognition Using Flexible Analytic Wavelet Transform from EEG Signals. IEEE Sens. J. 2019, 19, 2266–2274. [Google Scholar] [CrossRef]
Figure 1. V-A plane that defines four emotions with each subject’s self-assessment ratings and the subject-generic thresholds.
Figure 1. V-A plane that defines four emotions with each subject’s self-assessment ratings and the subject-generic thresholds.
Symmetry 11 00683 g001
Figure 2. Illustration of influence of the alteration of the training set.
Figure 2. Illustration of influence of the alteration of the training set.
Symmetry 11 00683 g002
Figure 3. The schemes for building subject-specific training set for (a) TRFE and (b) M-TRFE.
Figure 3. The schemes for building subject-specific training set for (a) TRFE and (b) M-TRFE.
Symmetry 11 00683 g003
Figure 4. Binary subject-specific classification accuracies on (a) arousal and (b) valence dimensions.
Figure 4. Binary subject-specific classification accuracies on (a) arousal and (b) valence dimensions.
Symmetry 11 00683 g004
Figure 5. Influence of M-TRFE feature elimination on binary classification: (a) arousal accuracy, (b) valence accuracy, (c) arousal f1 score, (d) valence f1 score, with different amounts of features eliminated.
Figure 5. Influence of M-TRFE feature elimination on binary classification: (a) arousal accuracy, (b) valence accuracy, (c) arousal f1 score, (d) valence f1 score, with different amounts of features eliminated.
Symmetry 11 00683 g005
Figure 6. Binary classification accuracy on arousal (a) and valence (b) dimensions under the feature selection paradigms of subject-specific, RFE and S-TRFE and M-TRFE.
Figure 6. Binary classification accuracy on arousal (a) and valence (b) dimensions under the feature selection paradigms of subject-specific, RFE and S-TRFE and M-TRFE.
Symmetry 11 00683 g006
Figure 7. Classification performances of three strategies for four emotions, (a) joy, (b) peace, (c) anger and (d) depression using separate binary classifiers.
Figure 7. Classification performances of three strategies for four emotions, (a) joy, (b) peace, (c) anger and (d) depression using separate binary classifiers.
Symmetry 11 00683 g007
Figure 8. Illustration of M-TRFE feature transferring: (a) the number of features that were eliminated for each subject, (b) the best results of subject-specific RFE using OvO, and (c) OA of M-TRFE when the number of subjects employed increases.
Figure 8. Illustration of M-TRFE feature transferring: (a) the number of features that were eliminated for each subject, (b) the best results of subject-specific RFE using OvO, and (c) OA of M-TRFE when the number of subjects employed increases.
Symmetry 11 00683 g008
Table 1. Notations of EEG features.
Table 1. Notations of EEG features.
Feature IndexNotations
44 EEG Power FeaturesAverage PSD in four bands for all channels.
16 EEG Power DifferencesDifference of average PSD in four bands for four channel pairs. (F4-F3, C4-C3, P4-P3 and O2-O1).
77 EEG Time Domain FeaturesMean, variance, zero crossing rate, Shannon entropy, spectral entropy, kurtosis and skewness of eleven channels.
Note: Eleven channels were sequential: F3, F4, Fz, C3, C4, Cz, P3, P4, Pz, O1 and O2. Four bands were theta (4–8 Hz), alpha (8–12 Hz), beta (12–30 Hz) and gamma (30–45 Hz).
Table 2. Pseudo codes of the algorithm for M-TRFE initialization.
Table 2. Pseudo codes of the algorithm for M-TRFE initialization.
Initialization of M-TRFE Algorithm
1Start initialization
2    for I = 1:s
3       for j= 1:f
4            Define V i = { x k , y k } using fth validating segment of subject i
5                Define J ˜ ( w , b , ζ k ) = 1 2 | | w | | 2 + 1 2 ( γ j ( i ) · k = 1 v i ζ k )
6                Train LSSVM model y j ( x ) = sign ( k = 1 v i α k y k x k x ) + b )
7     end for
8     Select the model and the regularization parameter γ 0 ( i ) by a cross-validation technique
9        forj1 = 1:s
10            ifj1 = s + 1
11                    j1 = 1
12             elsej1 = i
13             end if
14             Define cross-subject data V j 1 = { x k , y k } from working segment of subject j1
15             Define J ˜ ( w , b , ζ k ) = 1 2 | | w | | 2 + 1 2 ( γ 0 ( i ) · k = 1 v j 1 ζ k ) and train the model
16             Test model with the validating segment V i = { x k 1 , y k 1 } from subject i
17            Create subject ranking vector A H = A i A H
18             end for
19        Rank the most trusted subjects through ranking A H
20      end for
21End initialization
Table 3. Pseudo codes of the algorithm for M-TRFE feature ranking.
Table 3. Pseudo codes of the algorithm for M-TRFE feature ranking.
Feature Ranking of M-TRFE Algorithm
1Start feature ranking
2    for i = 1:s
3        Load V i = { x k , y k } , γ 0 ( i )
4        Calculate V H for a certain emotion and create blank space S i =
5        for j = 1: N o i
6            if H P ( x j ) < 0
7                 S i = S i x j
8            else S i = S i
9            end if
10            for j = 1:L
11                Build the O i = V i S i used for transferring task
12                Define J ˜ ( w , b , ζ k ) = 1 2 | | w | | 2 + 1 2 ( γ 0 ( i ) · k = 1 O i ζ k )
13                Find support vector w = k = 1 O i α k y k x k
14                for r = 1:L
15                     w ˜ ( r ) = f a [ | | w ( r ) | | 2 ] , D ˜ ( r ) = f a [ d H ( r ) + d L ( r ) ] , Δ Φ ˜ ( r ) = λ 1 w ˜ ( r ) + λ 2 D ˜ ( r )
16                end for
17             Create a blank feature ranking set R =
18             R ( j ) = R ( j ) arg min Δ Φ ˜
19               Eliminate R from feature set S
20         end for
21        Return feature ranking set S = J = 1 L R ( j )
22End feature ranking
Table 4. The most trusted subjects for binary classification.
Table 4. The most trusted subjects for binary classification.
Most Trusted SubjectArousalValence
1169
247
3320
4615
51518
Note: Subject No. 16 contributed his credible features to the transferring set in M-TRFE for arousal. Subject No. 9 contributed his credible features to the transferring set in M-TRFE for valence.
Table 5. The worst features and corresponding physiological significance ranked for binary classification.
Table 5. The worst features and corresponding physiological significance ranked for binary classification.
Worst FeatureArousalCorresponding Physiological SignificanceValenceCorresponding Physiological Significance
161CZ, β, PSD69Fz, γ, PSD
266O2, β, PSD62P3, β, PSD
352P4, α, PSD66O2, β, PSD
4129FZ, Zero-crossing rate61CZ, β, PSD
563P4, β, PSD64Pz, α, PSD
665O1, β, PSD70C3, γ, PSD
762P3, β, PSD129FZ, Zero-crossing rate
867F3, γ, PSD65O1, β, PSD
9127F3, Zero-crossing rate71C4, γ, PSD
1058FZ, β, PSD58FZ, β, PSD
Table 6. Binary classification performances of different cross-subject feature selections and SS methods.
Table 6. Binary classification performances of different cross-subject feature selections and SS methods.
Classification SchemeIndex
Mean Accuracy-ArousalMean Accuracy-ValenceMean F1 Score-ArousalMean F1 Score -Valence
Direct Scheme0.5089 (0.0257)0.5506 (0.0467)0.4961 (0.2701)0.4818 (0.0363)
S-TRFE0.6470 (0.0740)0.6875 (0.0588)0.6163 (0.0245)0.6838 (0.0489)
M-TRFE0.6494 (0.0496)0.6898 (0.0676)0.6571 (0.0513)0.6773 (0.0363)
G-TRFE0.5580 (0.0801)0.5680 (0.0696)0.5055 (0.0166)0.5361 (0.0482)
SS0.6549 (0.0701)0.6865 (0.1581)0.5364 (0.2864)0.6389 (0.1816)
Note: All classification performances listed above adopted an optimal number of transferring features. The subject-specific average value of accuracy is shown. The values in the brackets are the corresponding standard deviation.
Table 7. Worst features ranked for multiclass and corresponding physiological significance in multi-classification.
Table 7. Worst features ranked for multiclass and corresponding physiological significance in multi-classification.
Worst FeatureJoyPeaceAngerDepressionMutual RankingCorresponding Physiological Significance
161132636963P4, β, PSD
25866615261CZ, β, PSD
36663526652P4, α, PSD
463521296366O2, β, PSD
5134616661132CZ, Zero-crossing rate
6127655067127F3, Zero-crossing rate
71291275162129FZ, Zero-crossing rate
8137671277169FZ, γ, PSD
9132717113258FZ, β, PSD
105249675167F3, γ, PSD
Table 8. Most trusted subjects selected in multi classification.
Table 8. Most trusted subjects selected in multi classification.
Most Trusted SubjectJoyPeaceAngerDepressionMutual
13121231431
2159263221
3111331723
4163042414
582562126
Note: Subject No. 31 and 21 contributed their credible features to the transferring set in multiclass M-TRFE.
Table 9. Indexes of multiclass classification performance using OvO structure.
Table 9. Indexes of multiclass classification performance using OvO structure.
SSS-TRFEM-TRFEG-TRFE
OA0.59080.53420.65130.6205
Kappa Value0.42120.31820.46650.3016
ANOVA-p < 0.05p < 0.05p < 0.05
PrecisionJoy0.54760.39720.64160.4027
Peace0.75510.34750.74890.5212
Anger0.57460.34260.61460.5281
Depression0.71290.36880.88910.4938
RecallJoy0.5710.48220.57950.4121
Peace0.49000.30490.51730.3577
Anger0.79880.29780.85980.8544
Depression0.33640.46100.27230.2174
F1 ScoreJoy0.55910.43560.6090.4073
Peace0.59430.32480.61190.4242
Anger0.66840.31860.71680.6527
Depression0.45710.40980.41690.3019
Table 10. Comparison of our recognition approach with some other recent studies.
Table 10. Comparison of our recognition approach with some other recent studies.
StudyFeature Selection MethodClassifierIf or Not Cross Subject?OA
BinaryMulticlass
Koelstra, 2012 [36]-SVMNo0.6235-
Naser, 2013 [40]SVDSVMNo0.6525-
Zhu, 2014 [41]-SVMNo0.5795-
Li, 2015 [24]-DBNNo0.5130-
Atkinson, 2015 [30]mRMRSVMNo0.6151-
Chen, 2016 [25]CCASVMNo0.6040-
Shahnaz, 2016 [26]PCASVMNo0.6561-
Feradov, 2014 [42]-SVMNo-0.6200
Candra, 2011 [43]-SVMNo-0.6090
Nakisa, 2018 [44]ECPNNNo-0.6408–0.7085
Gupta, 2018 [45]FAWTRandom ForestYes-0.7143
Yin, 2017 [16]T-RFELSSVMYes0.56300.6205
Our workM-TRFELSSVMYes0.66950.6513
Note: The result of Nakisa’s EC-PNN had a deviation of ±0.0338. The result of Gupta’s work was the average accuracy of six channels (T7, T8, CP5, CP6, P7, P8). The result of Yin’s T-RFE was applied under our feature extraction.

Share and Cite

MDPI and ACS Style

Cai, J.; Chen, W.; Yin, Z. Multiple Transferable Recursive Feature Elimination Technique for Emotion Recognition Based on EEG Signals. Symmetry 2019, 11, 683. https://doi.org/10.3390/sym11050683

AMA Style

Cai J, Chen W, Yin Z. Multiple Transferable Recursive Feature Elimination Technique for Emotion Recognition Based on EEG Signals. Symmetry. 2019; 11(5):683. https://doi.org/10.3390/sym11050683

Chicago/Turabian Style

Cai, Jiahui, Wei Chen, and Zhong Yin. 2019. "Multiple Transferable Recursive Feature Elimination Technique for Emotion Recognition Based on EEG Signals" Symmetry 11, no. 5: 683. https://doi.org/10.3390/sym11050683

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop