Abstract
Micro-expressions are subtle facial movements that expose a person’s hidden emotions. Recognizing the micro-expression has importance for example in criminal investigations and psychotherapy. Compared with the shallower-architecture model, image magnification of these movements, which is also crucial for accurate recognition, has received relatively less attention in the field of micro-expression recognition. In this work, we find that there are some limitations during the training process, in particular, an imbalance in the distribution of motion amplitudes of samples, optical flow features, and semantic features. To mitigate their adverse effects, we propose adaptive balanced magnification, the balance of optical flow features and the balance of enhanced semantic features, to reduce these imbalances. Experimental results from three benchmarks (CASMEII, SAMM, and SMIC) show that our proposed method has higher accuracy and better recognition success than other micro-expression recognition methods.
Similar content being viewed by others
References
Ekman, P.: Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage, Revised WW Norton & Company, New York (2009)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: Continual prediction with lstm. Neural comput 12(10), 2451–2471 (2000)
Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. (TOG) 31(4), 1–8 (2012)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision, pp. 818–833. Springer. (2014)
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
Pang, J., Chen, K., Shi, J., Feng, H., Ouyang, W., and Lin, D.: Libra r-cnn: towards balanced learning for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 821–830 (2019)
Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., Xiaolan, F.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2015)
Karmakar, D., Sarkar, R., Datta, M.: Spoofed replay attack detection by multidimensional fourier transform on facial micro-expression regions. Signal Process. Image Commun. 93(7), 116164 (2021)
Liu, K.H., Jin, Q.S., Xu, H.C., Gan, Y.S., Liong, S.T.: Micro-expression recognition using advanced genetic algorithm. Signal Process. Image Commun. 93(3), 116153 (2021)
Ren, Z., Ning, H.: A review of micro-expression recognition research (in Chinese). Comput. Eng. Appl. 57(1), 38–47 (2021)
Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)
Li, X., Yu, J., Zhan, S.: Spontaneous facial micro-expression detection based on deep learning. In: 2016 IEEE 13th International Conference on Signal Processing (ICSP), pp. 1130–1134. IEEE. (2016)
Chaudhry, R., Ravichandran, A., Hager, G., Vidal, R.: Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1932–1939. IEEE. (2009)
Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more: micro-expression recognition from video using apex frame. Signal Process. Image Commun. 62, 82–92 (2018)
Chongyang, W., Min, P., Tao, B., Tong, C.: Micro-attention for micro-expression recognition. Neurocomputing 410, 354–362 (2020)
Liu, Y., Du, H., Zheng, L., Gedeon, T.: A neural micro-expression recognizer. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–4. IEEE. (2019)
Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., Xiaolan, F.: Casme II: an improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1), e86041 (2014)
Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M.: A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (fg), pp. 1–6. IEEE. (2013)
Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2016)
Davison, A.K., Merghani, W., Yap, M.H.: Objective classes for micro-facial expression recognition. J. Imaging. 4(10), 119 (2018)
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE. (2010)
Xia, Z., Hong, X., Gao, X., Feng, X., Zhao, G.: Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans. Multimed. 22(3), 626–640 (2019)
Liu, C., et al.: Beyond pixels: exploring new representations and applications for motion analysis. PhD thesis, Massachusetts Institute of Technology (2009)
King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. IEEE. (2009)
Gan, Y.S., Liong, S.-T., Yau, W.-C., Huang, Y.-C., Tan, L.-K.: Off-apexnet on micro-expression recognition system. Signal Process. Image Commun. 74, 129–139 (2019)
Van Quang, N., Chun, J., Tokuyama, T.: Capsulenet for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–7. IEEE. (2019)
Liong, S.-T., Gan, Y.S., See, J., Khor, H.-Q., and Huang, Y.-C.: Shallow triple stream three-dimensional CNN (STSTNET) for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5. IEEE. (2019)
Zhou, L., Mao, Q., Xue, L.: Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5. IEEE. (2019)
Krizhevsky, Alex., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In NIPS, page 1097–1105, 2012
Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with \(50\times\) fewer parameters and \(<0.5\) mb model size. arXiv preprint arXiv:1602.07360 (2016)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Acknowledgements
This work is supported by the National Natural Science Foundation of China (61872042, 61972375), the Key Project of Beijing Municipal Commission of Education (KZ201911417048), the Major Project of Technological Innovation 2030—“New Generation Artificial Intelligence” (2018AAA0100800), Premium Funding Project for Academic Human Resources Development in Beijing Union University (BPHR2020AZ01, BPH2020EZ01), Scientific Research Project of Beijing Municipal Commission of Education (KM202111417009), innovation funding project for postgraduates of Beijing Union University (YZ2020K001).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by B. K Bao.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhang, R., He, N., Wu, Y. et al. To balance: balanced micro-expression recognition. Multimedia Systems 28, 335–345 (2022). https://doi.org/10.1007/s00530-021-00842-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00530-021-00842-1