Abstract
Emotion expression encompasses various types of information, including face and eye movement, voice and body motion. Emotions collected from real conversations are difficult to classify using one channel. That is why multimodal techniques have recently become more popular in automatic emotion recognition. Multimodal databases that include audio, video, 3D motion capture and physiology data are quite rare. We collected The Russian Acted Multimodal Affective Set (RAMAS) − the first multimodal corpus in Russian language. Our database contains approximately 7 h of high-quality close-up video recordings of faces, speech, motion-capture data and such physiological signals as electro-dermal activity and photoplethysmogram. The subjects were 10 actors who played out interactive dyadic scenarios. Each scenario involved one of the basic emotions: Anger, Sadness, Disgust, Happiness, Fear or Surprise, and such characteristics of social interaction like Domination and Submission. In order to note emotions that subjects really felt during the process we asked them to fill in short questionnaires (self-reports) after each played scenario. The records were marked by 21 annotators (at least five annotators marked each scenario). We present our multimodal data collection, annotation process, inter-rater agreement analysis and the comparison between self-reports and received annotations. RAMAS is an open database that provides research community with multimodal data of faces, speech, gestures and physiology interrelation. Such material is useful for various investigations and automatic affective systems development.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Consensys GSR development kit. http://www.shimmersensing.com/products/gsr-optical-pulse-development-kit
Kinect v. 2. http://www.microsoft.com/en-us/kinectforwindows
Neurodata Lab LLC. http://www.neurodatalab.com
RAMAS scripts. http://neurodatalab.com/upload/technologies_files/scenarios/Scripts_RAMAS.pdf
RAMAS database (2016). http://neurodatalab.com/en/projects/RAMAS
Anderson, A., Hsiao, T., Metsis, V.: Classification of emotional arousal during multimedia exposure. In: Proceedings of the 10th International Conference on Pervasive Technologies Related to Assistive Environments, pp. 181–184. ACM (2017)
Ayvaz, U., Gürüler, H., Devrim, M.O.: Use of facial emotion recognition in e-learning systems. Inf. Technol. Learn. Tools 60(4), 95–104 (2017)
Bänziger, T., Pirker, H., Scherer, K.: GEMEP-GEneva multimodal emotion portrayals: a corpus for the study of multimodal emotional expressions. In: Proceedings of LREC, vol. 6, pp. 15–19 (2006)
Busso, C., Bulut, M., Lee, C.C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J.N., Lee, S., Narayanan, S.S.: IEMOCAP: interactive emotional dyadic motion capture database. Lang. Resour. Eval. 42(4), 335 (2008)
Chaw, T.V., Khor, S.W., Lau, P.Y.: Facial expression recognition using correlation of eyes regions. In: The FICT Colloquium 2016, p. 34, December 2016
De Silva, L.C., Miyasato, T., Nakatsu, R.: Facial emotion recognition using multi-modal information. In: Proceedings of 1997 International Conference on Information, Communications and Signal Processing, ICICS 1997, vol. 1, pp. 397–401. IEEE (1997)
D’mello, S.K., Kory, J.: A review and meta-analysis of multimodal affect detection systems. ACM Comput. Surv. 47(3), 43:1–43:36 (2015). http://doi.acm.org/10.1145/2682899
Douglas, M.: Purity and danger: an analysis of pollution and taboo London (1966)
El Ayadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recogn. 44(3), 572–587 (2011)
Gouizi, K., Bereksi Reguig, F., Maaoui, C.: Emotion recognition from physiological signals. J. Med. Eng. Technol. 35(6–7), 300–307 (2011)
Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Meas. 1(1), 77–89 (2007)
Karg, M., Samadani, A.A., Gorbet, R., Kühnlenz, K., Hoey, J., Kulić, D.: Body movements for affective expression: a survey of automatic recognition and generation. IEEE Trans. Affect. Comput. 4(4), 341–359 (2013)
Krippendorff, K.: Estimating the reliability, systematic error and random error of interval data. Educ. Psychol. Meas. 30(1), 61–70 (1970)
Mayer, J.D., Salovey, P., Caruso, D.R., Sitarenios, G.: Measuring emotional intelligence with the MSCEIT V2. 0. Emotion 3(1), 97 (2003)
Metallinou, A., Lee, C.C., Busso, C., Carnicke, S., Narayanan, S.: The USC creativeIT database: a multimodal database of theatrical improvisation. Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, p. 55 (2010)
Rachman, S.: Anxiety. Psychology Press Ltd., Publishers, East Sussex (1998)
Ranganathan, H., Chakraborty, S., Panchanathan, S.: Multimodal emotion recognition using deep learning architectures. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–9, March 2016. https://doi.org/10.1109/WACV.2016.7477679
Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–8. IEEE (2013)
Russell, J.A., Fernández-Dols, J.M.: The Psychology of Facial Expression. Cambridge University Press, Cambridge (1997)
Sergienko, E.G., Vetrova, I.I., Volochkov, A.A., Popov, A.Y.: Adaptation of J. Mayer P. Salovey and D. Caruso emotional intelligence test on russian-speaking sample. Psikhologicheskii Zhurnal 31(1), 55–73 (2010)
Sloetjes, H., Wittenburg, P.: Annotation by category: ELAN and ISO DCR. In: LREC (2008)
Tarnowski, P., Kołodziej, M., Majkowski, A., Rak, R.J.: Emotion recognition using facial expressions. Procedia Comput. Sci. 108, 1175–1184 (2017)
Tomkins, S.: Affect Imagery Consciousness: Volume II: The Negative Affects. Springer, New York (1963)
Volkova, E., De La Rosa, S., Bülthoff, H.H., Mohler, B.: The MPI emotional body expressions database for narrative scenarios. PloS one 9(12), e113647 (2014)
Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., André, E.: The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 831–834. ACM (2013)
Acknowledgements
Supported by Neurodata Lab LLC. The authors would like to thank Elena Arkova for finding the actors and helping with the scenarios and experimental procedure, and Irina Vetrova for evaluating the emotional intelligence of the annotators with MSCEIT v 2.0 test.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Perepelkina, O., Kazimirova, E., Konstantinova, M. (2018). RAMAS: Russian Multimodal Corpus of Dyadic Interaction for Affective Computing. In: Karpov, A., Jokisch, O., Potapova, R. (eds) Speech and Computer. SPECOM 2018. Lecture Notes in Computer Science(), vol 11096. Springer, Cham. https://doi.org/10.1007/978-3-319-99579-3_52
Download citation
DOI: https://doi.org/10.1007/978-3-319-99579-3_52
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-99578-6
Online ISBN: 978-3-319-99579-3
eBook Packages: Computer ScienceComputer Science (R0)