CSAC-Net: Fast Adaptive sEMG Recognition through Attention Convolution Network and Model-Agnostic Meta-Learning
<p>An overview of the method in a block diagram representation.</p> "> Figure 2
<p>Two processed surface EMG samples and their STFT spectrogram (Hanning window, window length = 64).</p> "> Figure 3
<p>The detailed architecture of the proposed CSAC-Net.</p> "> Figure 4
<p>The Channel-Spatial Attention Module.</p> "> Figure 5
<p>The structure of Channel Attention Module.</p> "> Figure 6
<p>The structure of Spatial Attention Module.</p> "> Figure 7
<p>The structure of the CSAC-Cell.</p> "> Figure 8
<p>Specific block diagram of MAML framework in our work. The colorful lines represent the sEMG signals used for training and verification after windowing. The yellow star represents the model for new subjects.</p> "> Figure 9
<p>For different methods and datasets, the accuracy of 10 repetitions.</p> ">
Abstract
:1. Introduction
- To mitigate the effect of individual differences, CSAC-Net is used to update the model based on MAML. Compared with the traditional learning method, it focuses on improving the overall learning ability of the model, rather than the ability to solve a specific sEMG classification problem.
- To meet the challenge of model complexity and feature extraction, we combine the lite CNN network with the attention mechanism. By analyzing the spectrogram in the time-frequency domain of multi-channel sEMG signal after preprocessing, the features of sEMG signal are effectively extracted, which contributes to better performance than utilizing raw sEMG data in the time domain.
- CSAC-Net requires the maximization of the sensitivity of loss functions of new tasks with respect to the parameters when we are training the model. Minimal parameter changes can bring great improvement to the model. In this way, it is possible to quickly adapt to new tasks with only a small amount of data by gradient adjustment on the basis of initial parameters.
- In order to demonstrate the generalization and fast adaptation of the classification of our model, three datasets are selected to carry out the experiments. Our model achieves better performance than previous work.
2. Related Works
2.1. Traditional Methods for Gesture Recognition through sEMG
2.2. Deployment of Attention Mechanism in Biomedical Signal Decoding
2.3. Current Progress in Domain Adaptation
3. Methods and Theories
3.1. Data Preprocessing
3.2. Feature Extraction
3.3. The Architecture of CSAC-Net
3.3.1. Channel-Spatial Attention Module
3.3.2. CSAC-Cell
3.3.3. Loss Function
3.4. Evaluating Indicator: N-Way K-Shot
3.5. MAML Framework
Algorithm 1 MAML-training |
Require: : distribution over tasks Require: , : step size hyperparameters Require: Iteration number of epoch model 1: randomly initialize 2: while not epoch do 3: Sample batch of tasks 4: for all do 5: Sample datapoints from 6: Evaluate with respect to NK examples 7: Evaluate 8: Evaluate 9: end for 10: Update 11: end while |
Algorithm 2 MAML-testing |
Require: training data new task T Require: learned 1: Evaluate 2: Compute adapted parameters with gradient descent |
4. Experiments and Results
4.1. Dataset
4.2. Basic Experiments
4.3. Ablation Studies
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
sEMG | surface electromyography |
STFT | short time Fourier transform |
MAML | model-agnostic meta-learning |
CSAC-Net | Channel-Spatial Attention Convolution Network |
FFT | fast Fourier transform |
GAP | global average pooling |
GWP | global max pooling |
MAV | mean average value |
References
- Stefanou, T.; Guiraud, D.; Fattal, C.; Azevedo-Coste, C.; Fonseca, L. Frequency-Domain sEMG Classification Using a Single Sensor. Sensors 2022, 22, 1939. [Google Scholar] [CrossRef] [PubMed]
- Hu, J.; Hou, Z.-G.; Chen, Y.-X.; Zhang, F.; Wang, W.-Q. Lower limb rehabilitation robots and in-teractive control methods. Acta Autom. Sin. 2014, 40, 2377–2390. [Google Scholar]
- Nasri, N.; Orts-Escolano, S.; Cazorla, M. An sEMG-Controlled 3D Game for Rehabilitation Therapies: Real-Time Time Hand Gesture Recognition Using Deep Learning Techniques. Sensors 2020, 20, 6451. [Google Scholar] [CrossRef] [PubMed]
- Ding, Q.-C.; Xiong, A.-B.; Zhao, X.-G.; Han, J.-D. A review on researches and applications of sEMG-based motion intent recognition methods. Acta Autom. Sin. 2016, 42, 13–25. [Google Scholar]
- Li, Z.-Y.; Zhao, X.-G.; Zhang, B.; Ding, Q.-C.; Zhang, D.-H.; Han, J.-D. Review of sEMG-based motion intent recognition methods in non-ideal conditions. Acta Autom. Sin. 2021, 47, 955–969. [Google Scholar]
- Tigrini, A.; Pettinari, L.A.; Verdini, F.; Fioretti, S.; Mengarelli, A. Shoulder Motion Intention Detection Through Myoelectric Pattern Recognition. IEEE Sens. Lett. 2021, 5, 6001904. [Google Scholar] [CrossRef]
- Wang, Y.; Tian, Y.; Zhu, J.; She, H.; Yokoi, H.; Jiang, Y.; Huang, Q. A Study on the Classification Effect of sEMG Signals in Different Vibration Environments Based on the LDA Algorithm. Sensors 2021, 21, 6234. [Google Scholar] [CrossRef]
- Toledo-Pérez, D.C.; Rodríguez-Reséndiz, J.; Gómez-Loenzo, R.A.; Jauregui-Correa, J.C. Support Vector Machine-Based EMG Signal Classification Techniques: A Review. Appl. Sci. 2019, 9, 4402. [Google Scholar] [CrossRef] [Green Version]
- Siu, H.C.; Shah, J.A.; Stirling, L.A. Classification of Anticipatory Signals for Grasp and Release from Surface Electromyography. Sensors 2016, 16, 1782. [Google Scholar] [CrossRef] [Green Version]
- Tsinganos, P.; Cornelis, B.; Cornelis, J.; Jansen, B.; Skodras, A. Improved Gesture Recognition Based on sEMG Signals and TCN. In Proceedings of the ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; pp. 1169–1173. [Google Scholar]
- Chen, L.; Fu, J.; Wu, Y.; Li, H.; Zheng, B. Hand Gesture Recognition Using Compact CNN via Surface Electromyography Signals. Sensors 2020, 20, 672. [Google Scholar] [CrossRef] [Green Version]
- Campbell, E.; Phinyomark, A.; Al-Timemy, A.H.; Khushaba, R.N.; Petri, G.; Scheme, E. Differences in EMG Feature Space between Able-Bodied and Amputee Subjects for Myoelectric Control. In Proceedings of the 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER), San Francisco, CA, USA, 20–23 March 2019; pp. 33–36. [Google Scholar]
- Kim, K.-T.; Guan, C.; Lee, S.-W. A Subject-Transfer Framework Based on Single-Trial EMG Analysis Using Convolutional Neural Networks. IEEE Trans. Neural Syst. Rehabil. Eng. 2020, 28, 94–103. [Google Scholar] [CrossRef]
- Pan, L.; Crouch, D.L.; Huang, H. Myoelectric Control Based on a Generic Musculoskeletal Model: Toward a Multi-User Neural-Machine Interface. IEEE Trans. Neural Syst. Rehabil. Eng. 2018, 26, 1435–1442. [Google Scholar] [CrossRef] [PubMed]
- Tsinganos, P.; Cornelis, J.; Cornelis, B.; Jansen, B.; Skodras, A. Transfer Learning in sEMG-based Gesture Recognition. In Proceedings of the 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA), Chania Crete, Greece, 12–14 July 2021; pp. 1–7. [Google Scholar]
- Ashish, V.; Noam, S.; Niki, P.; Jakob, U.; Llion, J.; Aidan, N.G.; Łukasz, K.; Illia, P. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 6000–6010. [Google Scholar]
- Finn, C.; Abbeel, P.; Levine, S. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017; Volume 70, pp. 1126–1135. [Google Scholar]
- Khezrik, M.; Jahed, M. A Neuro–Fuzzy Inference System for sEMG-Based Identification of Hand Motion Commands. IEEE Trans. Ind. Electron. 2011, 58, 1952–1960. [Google Scholar] [CrossRef]
- Triwiyanto, T.; Caesarendra, W.; Purnomo, M.H.; Sułowicz, M.; Wisana, I.D.G.H.; Titisari, D.; Lamidi, L.; Rismayani, R. Embedded Machine Learning Using a Multi-Thread Algorithm on a Raspberry Pi Platform to Improve Prosthetic Hand Performance. Micromachines 2022, 13, 191. [Google Scholar] [CrossRef]
- Leserri, D.; Grimmelsmann, N.; Mechtenberg, M.; Meyer, H.G.; Schneider, A. Evaluation of sEMG Signal Features and Segmentation Parameters for Limb Movement Prediction Using a Feedforward Neural Network. Mathematics 2022, 10, 932. [Google Scholar] [CrossRef]
- Khushaba, R.N.; Nazarpour, K. Decoding HD-EMG Signals for Myoelectric Control - How Small Can the Analysis Window Size be? IEEE Robot. Autom. Lett. 2021, 6, 8569–8574. [Google Scholar] [CrossRef]
- Adewuyi, A.; Hargrove, L.; Kuiken, T.A. Evaluating EMG feature and classifier selection for application to partial-hand prosthesiscontrol. Front. Neurorobot. 2016, 10, 15. [Google Scholar] [CrossRef] [Green Version]
- Wu, H.; Dyson, M.; Nazarpour, K. Arduino-Based Myoelectric Control: Towards Longitudinal Study of Prosthesis Use. Sensors 2021, 21, 763. [Google Scholar] [CrossRef]
- Coskun, M.; Yildirim, O.; Demir, Y.; Acharya, U.R. Efficient deep neural network model for classification of grasp types using sEMG signals. J. Ambient. Intell. Humaniz. Comput. 2021, 1–14. [Google Scholar] [CrossRef]
- Shi, X.; Qin, P.; Zhu, J.; Zhai, M.; Shi, W. Feature Extraction and Classification of Lower Limb Motion Based on sEMG Signals. IEEE Access 2020, 8, 132882–132892. [Google Scholar] [CrossRef]
- Mnih, V.; Heess, N.; Graves, A. Recurrent models of visual attention. Adv. Neural Inf. Process. Syst. 2014, 27, 2204–2212. [Google Scholar]
- Tao, W.; Li, C.; Song, R.; Cheng, J.; Liu, Y.; Wan, F.; Chen, X. EEG-based emotion recognition via channel-wise attention and self-attention. IEEE Trans. Affect. Comput. 2020. [Google Scholar] [CrossRef]
- Woo, S.; Park, J.; Lee, J.-Y.; Kweon, I.S. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar]
- Chen, G.; Wang, W.; Wang, Z.; Liu, H.; Zang, Z.; Li, W. Two-dimensional discrete feature based spatial attention CapsNet For sEMG signal recognition. Appl. Intell. 2020, 50, 3503–3520. [Google Scholar] [CrossRef]
- Zhang, Y.; Chen, S.; Cao, W.; Guo, P.; Gao, D.; Wang, M.; Zhou, J.; Wang, T. MFFNet: Multi-dimensional Feature Fusion Network based on attention mechanism for sEMG analysis to detect muscle fatigue. Expert Syst. Appl. 2021, 185, 115639. [Google Scholar] [CrossRef]
- Du, Y.; Jin, W.; Wei, W.; Hu, Y.; Geng, W. Surface EMG-Based Inter-Session Gesture Recognition Enhanced by Deep Domain Adaptation. Sensors 2017, 17, 458. [Google Scholar] [CrossRef] [Green Version]
- Guangjun, Y.A.N.; Wanzhong, C.H.E.N.; Tao, Z.H.A.N.G.; Yun, J.I.A.N.G.; Shuifang, R.E.N. Research on Gesture Classification Methods in Amputee Subjects Based on Gray Theory Model. J. Electron. Inf. Technol. 2021, 43, 2552–2560. [Google Scholar]
- Rahimian, E.; Zabihi, S.; Asif, A.; Atashzar, S.F.; Mohammadi, A. Few-Shot Learning for Decoding Surface Electromyography for Hand Gesture Recognition. In Proceedings of the ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada, 13 May 2021; pp. 1300–1304. [Google Scholar]
- Banluesombatkul, N.; Ouppaphan, P.; Leelaarporn, P.; Lakhan, P.; Chaitusaney, B.; Jaimchariyatam, N.; Chuangsuwanich, E.; Chen, W.; Phan, H.; Dilokthanakul, N.; et al. MetaSleepLearner: A Pilot Study on Fast Adaptation of Bio-Signals-Based Sleep Stage Classifier to New Individual Subject Using Meta-Learning. IEEE J. Biomed. Health Inform. 2021, 25, 1949–1963. [Google Scholar] [CrossRef]
- Wei, W.; Wong, Y.; Du, Y.; Hu, Y.; Kankanhalli, M.; Geng, W. A Multi-stream Convolutional Neural Network for sEMG-based Gesture Recognition in Muscle-computer Interface. Pattern Recognit. Lett. 2019, 119, 131–138. [Google Scholar] [CrossRef]
- Geng, W.; Du, Y.; Jin, W.; Wei, W.; Hu, Y.; Li, J. Gesture Recognition by Instantaneous Surface EMG Images. Sci. Rep. 2016, 6, 36571. [Google Scholar] [CrossRef]
- Chen, L.; Zhang, H.; Xiao, J.; Nie, L.; Shao, J.; Liu, W.; Chua, T. SCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 6298–6306. [Google Scholar]
- Vinyals, O.; Blundell, C.; Lillicrap, T.; Kavukcuoglu, K. Matching networks for one shot learning. In Proceedings of the 30th In-ternational Conference on Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016; pp. 3637–3645. [Google Scholar]
- Arnold, S.M.; Mahajan, P.; Datta, D.; Bunner, I.; Zarkias, K.S. learn2learn: A Library for Meta-Learning Research. arXiv 2020, arXiv:2008.12284. [Google Scholar]
- Chen, J.; Bi, S.; Zhang, G.; Cao, G. High-Density Surface EMG-Based Gesture Recognition Using a 3D Convolutional Neural Network. Sensors 2020, 20, 1201. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Lai, Z.; Kang, X.; Wang, H.; Zhang, W.; Zhang, X.; Gong, P.; Niu, L.; Huang, H. STCN-GR: Spatial-Temporal Convolutional Networks for Surface-Electromyography-Based Gesture Recognition. In Proceedings of the International Conference on Neural Information Processing, Sanur Bali, Indonesia, 8–12 December 2021; pp. 27–39. [Google Scholar]
- Zhang, Y.; Chen, Y.; Yu, H.; Yang, X.; Lu, W. Learning Effective Spatial–Temporal Features for sEMG Armband-Based Gesture Recognition. IEEE Internet Things J. 2020, 7, 6979–6992. [Google Scholar] [CrossRef]
- Padhy, S. A tensor-based approach using multilinear SVD for hand gesture recognition from SEMG signals. IEEE Sens. J. 2020, 21, 6634–6642. [Google Scholar] [CrossRef]
- Betthauser, J.L.; Krall, J.T.; Bannowsky, S.G.; Lévay, G.; Kaliki, R.R.; Fifer, M.S.; Thakor, N.V. Stable Responsive EMG Sequence Prediction and Adaptive Reinforcement With Temporal Convolutional Networks. IEEE Trans. Biomed. Eng. 2020, 67, 1707–1717. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Cover, T.; Hart, P. Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 1967, 13, 21–27. [Google Scholar] [CrossRef]
- Amari, S.; Wu, S. Improving support vector machine classifiers by modifying kernel functions. Neural Netw. 1999, 12, 783–789. [Google Scholar] [CrossRef]
- Breiman, L. Classification and Regression Trees; Routledge: New York, NY, USA, 1984. [Google Scholar]
Input | DB-a | DB-b | DB-c |
---|---|---|---|
Raw sEMG | 46.18 | 52.81 | 42.59 |
FFT Spectrum | 67.71 | 73.13 | 67.13 |
STFT Spectrogram | 94.10 | 94.06 | 94.44 |
Methods | DB-a | DB-b | DB-c |
---|---|---|---|
STFT Spectrogram + CSAC-Net | 94.10 | 94.06 | 94.44 |
STFT Spectrogram + Resnet18 [45] | 91.32 | 87.81 | 89.81 |
MAV + ED-TCN [44] | 93.75 | 91.88 | 90.28 |
MAV + KNN (k = 3) [46] | 88.54 | 88.75 | 91.67 |
MAV + SVM ( kernal: Gaussian ) [47] | 81.94 | 86.88 | 85.69 |
MAV + Tree [48] | 68.06 | 60.63 | 77.70 |
Dataset | 1Shot-Top1 | 1Shot-Top5 | 5Shot-Top1 | 5Shot-Top5 |
---|---|---|---|---|
DB-a | 72.50 | 65.00 | 77.50 | 76.49 |
DB-b | 62.50 | 57.50 | 83.00 | 81.00 |
DB-c | 80.00 | 70.00 | 83.50 | 80.99 |
Dataset | 1Shot-Top1 | 1Shot-Top5 | 5Shot-Top1 | 5Shot-Top5 |
---|---|---|---|---|
DB-a | 61.56 | 59.06 | 83.19 | 82.44 |
DB-b | 43.12 | 40.63 | 73.12 | 71.25 |
DB-c | 68.75 | 65.31 | 82.00 | 80.81 |
Methods | DB-a | DB-b | DB-c |
---|---|---|---|
STFT Spectrogram + CSAC-Net | 82.50 | 81.00 | 80.91 |
MLSVD + DL [43] | – | 75.40 | 68.30 |
STFT Spectrogram + Resnet18 [45] + MAML | 56.70 | 57.70 | 57.90 |
MAV + ED-TCN [44] + MAML | 28.40 | 30.15 | 29.30 |
raw data + extended AdaBN [31] | – | 55.30 | 35.10 |
input | Dataset | 1Shot-Top1 | 1Shot-Top5 | 5Shot-Top1 | 5Shot-Top5 |
---|---|---|---|---|---|
Raw sEMG | DB-a | 32.50 | 30.00 | 45.00 | 44.00 |
DB-b | 30.00 | 27.50 | 40.00 | 39.89 | |
DB-c | 37.50 | 35.00 | 45.00 | 44.00 | |
FFT | DB-a | 42.50 | 32.50 | 46.00 | 43.00 |
Specturm | DB-b | 40.00 | 37.50 | 44.50 | 43.50 |
DB-c | 42.50 | 40.00 | 38.50 | 38.00 |
input | Dataset | 1Shot-Top1 | 1Shot-Top5 | 5Shot-Top1 | 5Shot-Top5 |
---|---|---|---|---|---|
Raw sEMG | DB-a | 38.13 | 37.19 | 41.00 | 40.56 |
DB-b | 30.00 | 29.06 | 36.25 | 35.81 | |
DB-c | 45.00 | 43.44 | 41.25 | 40.26 | |
FFT | DB-a | 44.69 | 43.13 | 50.44 | 50.12 |
Specturm | DB-b | 32.81 | 31.56 | 40.31 | 39.94 |
DB-c | 36.25 | 34.38 | 43.69 | 43.31 |
Dataset | Validation Accuracy | Test Accuracy | Time for Training |
---|---|---|---|
DB-a | 94.85 | 23.75 | 253112 |
DB-b | 95.18 | 57.50 | 316119 |
DB-c | 95.83 | 35.83 | 189864 |
Meta Batch Size | Dataset | 1Shot | 5Shot |
---|---|---|---|
8 | DB-a | 4143 | 5316 |
DB-b | 4605 | 5515 | |
DB-c | 3885 | 5144 | |
64 | DB-a | 8234 | 17305 |
DB-b | 8547 | 17757 | |
DB-c | 8462 | 17295 |
Model | Dataset | 1Shot-Top1 | 1Shot-Top5 | 5Shot-Top1 | 5Shot-Top5 |
---|---|---|---|---|---|
SAC-Net | DB-a | 47.50 | 42.50 | 71.50 | 67.00 |
DB-b | 66.56 | 65.63 | 76.50 | 72.00 | |
DB-c | 55.00 | 50.00 | 80.50 | 77.50 | |
CAC-Net | DB-a | 50.00 | 40.00 | 63.50 | 62.50 |
DB-b | 50.00 | 47.50 | 63.00 | 61.50 | |
DB-c | 42.50 | 37.50 | 63.50 | 61.00 | |
CNN | DB-a | 40.00 | 32.50 | 67.00 | 65.50 |
DB-b | 52.50 | 42.50 | 72.50 | 71.50 | |
DB-c | 47.50 | 40.00 | 61.50 | 60.00 |
Model | Dataset | 1Shot-Top1 | 1Shot-Top5 | 5Shot-Top1 | 5Shot-Top5 |
---|---|---|---|---|---|
SAC-Net | DB-a | 41.25 | 38.44 | 82.75 | 81.50 |
DB-b | 42.50 | 42.19 | 69.19 | 67.94 | |
DB-c | 49.69 | 47.81 | 75.19 | 73.87 | |
CAC-Net | DB-a | 38.44 | 37.80 | 72.94 | 72.37 |
DB-b | 38.75 | 34.69 | 63.94 | 62.75 | |
DB-c | 39.69 | 37.81 | 62.62 | 62.31 | |
CNN | DB-a | 42.50 | 40.94 | 70.19 | 68.62 |
DB-b | 44.06 | 43.44 | 69.25 | 68.75 | |
DB-c | 43.13 | 42.19 | 65.31 | 64.94 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Fan, X.; Zou, L.; Liu, Z.; He, Y.; Zou, L.; Chi, R. CSAC-Net: Fast Adaptive sEMG Recognition through Attention Convolution Network and Model-Agnostic Meta-Learning. Sensors 2022, 22, 3661. https://doi.org/10.3390/s22103661
Fan X, Zou L, Liu Z, He Y, Zou L, Chi R. CSAC-Net: Fast Adaptive sEMG Recognition through Attention Convolution Network and Model-Agnostic Meta-Learning. Sensors. 2022; 22(10):3661. https://doi.org/10.3390/s22103661
Chicago/Turabian StyleFan, Xinchen, Lancheng Zou, Ziwu Liu, Yanru He, Lian Zou, and Ruan Chi. 2022. "CSAC-Net: Fast Adaptive sEMG Recognition through Attention Convolution Network and Model-Agnostic Meta-Learning" Sensors 22, no. 10: 3661. https://doi.org/10.3390/s22103661