Abstract
Developing virtual characters with naturalistic game playing capabilities is an increasingly researched topic in Human-Computer Interaction. Possible roles for such characters include virtual teachers, personal care assistants, and companions for children. Laughter is an under-investigated emotional expression both in Human-Human and Human-Computer Interaction. The EU Project ILHAIRE, aims to study this phenomena and endow machines with laughter detection and synthesis capabilities. The Laugh when you’re winning project, developed during the eNTERFACE 2013 Workshop in Lisbon, Portugal, aimed to set up and test a game scenario involving two human participants and one such virtual character. The game chosen, the yes/no game, induces natural verbal and non-verbal interaction between participants, including frequent hilarious events, e.g., one of the participants saying “yes” or “no” and so losing the game. The setup includes software platforms, developed by the ILHAIRE partners, allowing automatic analysis and fusion of human participants’ multimodal data (voice, facial expression, body movements, respiration) in real-time to detect laughter. Further, virtual characters endowed with multimodal skills were synthesised in order to interact with the participants by producing laughter in a natural way.
Chapter PDF
Similar content being viewed by others
References
Bachorowski, J., Smoski, M.J., Owren, M.J.: Automatic discrimination between laughter and speech. Journal of the Acoustical Society of America 110, 1581–1597 (2001)
Becker-Asano, C., Ishiguro, H.: Laughter in social robotics - no laughing matter. In: International Workshop on Social Intelligence Design (SID 2009), pp. 287–300 (2009)
Becker-Asano, C., Kanda, T., Ishi, C., Ishiguro, H.: How about laughter? perceived naturalness of two laughing humanoid robots. In: 3rd International Conference on Affective Computing & Intelligent Interaction, Affective Computing and Intelligent Interaction and Workshops, pp. 1–6 (2009)
Bernhardt, D., Robinson, P.: Detecting affect from non-stylised body motions. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 59–70. Springer, Heidelberg (2007)
Bourgeois, P., Hess, U.: The impact of social context on mimicry. Biological Psychology 77(3), 343–352 (2008)
Brand, M.: Voice puppetry. In: Proceedings of Conference on Computer Graphics and Interactive Techniques, pp. 21–28 (1999)
Bregler, C., Covell, M., Slaney, M.: Video rewrite: Driving visual speech with audio. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1997, pp. 353–360. ACM Press/Addison-Wesley Publishing Co., New York (1997), http://dx.org/10.1145/258734.258880
Cai, R., Lu, L., Zhang, H., Cai, L.: Highlight sound effects detection in audio stream. In: Proceedings of the 2003 IEEE International Conference on Multimedia and Expo (ICME). Baltimore, USA, pp. 37–40 (2003)
Castellano, G., Villalba, S.D., Camurri, A.: Recognising human emotions from body movement and gesture dynamics. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 71–82. Springer, Heidelberg (2007)
Cohen, M.M., Massaro, D.W.: Modeling coarticulation in synthetic visual speech. In: Models and Techniques in Computer Animation, pp. 139–156. Springer (1993)
Cosker, D., Edge, J.: Laughing, crying, sneezing and yawning: Automatic voice driven animation of non-speech articulations. In: Proceedings of Computer Animation and Social Agents (CASA 2009), pp. 21–24 (2009)
Deng, Z., Lewis, J., Neumann, U.: Synthesizing speech animation by learning compact speech co-articulation models. In: Computer Graphics International 2005, pp. 19–25 (2005)
DiLorenzo, P.C., Zordan, V.B., Sanders, B.L.: Laughing out loud: Control for modeling anatomically inspired laughter using audio. ACM Transactions on Graphics (TOG) 27(5), 125 (2008)
Ezzat, T., Geiger, G., Poggio, T.: Trainable videorealistic speech animation. In: Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 57–64 (2004)
Fukushima, S., Hashimoto, Y., Nozawa, T., Kajimoto, H.: Laugh enhancer using laugh track synchronized with the user’s laugh motion. In: CHI 2010 Extended Abstracts on Human Factors in Computing Systems, CHI EA 2010, pp. 3613–3618. ACM, New York (2010)
Gilroy, S.W., Cavazza, M., Niranen, M., Andre, E., Vogt, T., Urbain, J., Benayoun, M., Seichter, H., Billinghurst, M.: Pad-based multimodal affective fusion. In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, ACII 2009 (2009)
Gosling, S., Rentfrow, P.J., Swann, W.B.: A very brief measure of the big-five personality domains. Journal of Research in Personality 37(6), 504–528 (2003)
Hofmann, J., Platt, T., Urbain, J., Niewiadomski, R., Ruch, W.: Laughing avatar interaction evaluation form. Unpublished Research Instrument (2012)
Kennedy, L., Ellis, D.: Laughter detection in meetings. In: NIST ICASSP 2004 Meeting Recognition Workshop, pp. 118–121. Montreal (May 2004)
Kleinsmith, A., Bianchi-Berthouze, N.: Affective body expression perception and recognition: A survey (2012)
Kleinsmith, A., Bianchi-Berthouze, N., Steed, A.: Automatic recognition of non-acted affective postures. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 41(4), 1027–1038 (2011)
Knox, M.T., Mirghafori, N.: Automatic laughter detection using neural networks. In: Proceedings of Interspeech 2007, pp. 2973–2976. Antwerp, Belgium (2007)
Kshirsagar, S., Magnenat-Thalmann, N.: Visyllable based speech animation. Comput. Graph. Forum 22(3), 632–640 (2003)
Lasarcyk, E., Trouvain, J.: Imitating conversational laughter with an articulatory speech synthesis. In: Proceedings of the Interdisciplinary Workshop on The Phonetics of Laughter, pp. 43–48 (2007)
Leite, I., Castellano, G., Pereira, A., Martinho, C., Paiva, A.: Modelling empathic behaviour in a robotic game companion for children: An ethnographic study in real-world settings. In: Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction, pp. 367–374. ACM (2012)
Liu, W., Yin, B., Jia, X., Kong, D.: Audio to visual signal mappings with hmm. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2004 (2004)
Mancini, M., Glowinski, D., Massari, A.: Realtime expressive movement detection using the eyesweb xmi platform. In: Camurri, A., Costa, C. (eds.) INTETAIN. LNICST, vol. 78, pp. 221–222. Springer (2011)
Mancini, M., Hofmann, J., Platt, T., Volpe, G., Varni, G., Glowinski, D., Ruch, W., Camurri, A.: Towards automated full body detection of laughter driven by human expert annotation. In: Proceedings of the Fifth Biannual Humaine Association Conference on Affective Computing and Intelligent Interaction, Affective Interaction in Natural Environments (AFFINE) Workshop, Geneva, Switzerland, pp. 757–762 (2013)
Mancini, M., Varni, G., Glowinski, D., Volpe, G.: Computing and evaluating the body laughter index. Human Behavior Understanding, 90–98 (2012)
Meng, H., Kleinsmith, A., Bianchi-Berthouze, N.: Multi-score learning for affect recognition: The case of body postures. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.-C. (eds.) ACII 2011, Part I. LNCS, vol. 6974, pp. 225–234. Springer, Heidelberg (2011)
Niewiadomski, R., Mancini, M., Baur, T., Varni, G., Griffin, H., Aung, M.: MMLI: Multimodal multiperson corpus of laughter in interaction. In: Fourth Int. Workshop on Human Behavior Understanding, in Conjunction with ACM Multimedia 2013 (2013)
Niewiadomski, R., Pelachaud, C.: Towards multimodal expression of laughter. In: Nakano, Y., Neff, M., Paiva, A., Walker, M. (eds.) IVA 2012. LNCS, vol. 7502, pp. 231–244. Springer, Heidelberg (2012)
Niewiadomski, R., Hofmann, J., Urbain, J., Platt, T., Wagner, J., Piot, B., Cakmak, H., Pammi, S., Baur, T., Dupont, S., Geist, M., Lingenfelser, F., McKeown, G., Pietquin, O., Ruch, W.: Laugh-aware virtual agent and its impact on user amusement. In: Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems, AAMAS 2013. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, pp. 619–626 (2013)
Oura, K.: HMM-based speech synthesis system (HTS) (computer program webpage), http://hts.sp.nitech.ac.jp/ (consulted on June 22, 2011)
Owens, M.D.: It’s all in the game: Gamification, games, and gambling. Gaming Law Review and Economics 16 (2012)
Petridis, S., Pantic, M.: Audiovisual discrimination between laughter and speech. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Las Vegas, Nevada, pp. 5117–5120 (2008)
Petridis, S., Pantic, M.: Is this joke really funny? Judging the mirth by audiovisual laughter analysis. In: Proceedings of the IEEE International Conference on Multimedia and Expo, New York, USA, pp. 1444–1447 (2009)
Poe, E.A.: Maelzel’s chess-player. In: Southern Literary Messenger, vol. 2, pp. 318–326 (1836)
Qu, B., Pammi, S., Niewiadomski, R., Chollet, G.: Estimation of FAPs and intensities of AUs based on real-time face tracking. In: Pucher, M., Cosker, D., Hofer, G., Berger, M., Smith, W. (eds.) The 3rd International Symposium on Facial Analysis and Animation. ACM (2012)
Ruch, W., Ekman, P.: The expressive pattern of laughter. In: Kaszniak, A. (ed.) Emotion, Qualia and Consciousness, pp. 426–443. World Scientific Publishers, Tokyo (2001)
Ruch, W., Proyer, R.: Extending the study of gelotophobia: On gelotophiles and katagelasticists. Humor-International Journal of Humor Research 22(1/2), 183–212 (2009)
Ruf, T., Ernst, A., Küblbeck, C.: Face detection with the sophisticated high-speed object recognition engine (shore). In: Heuberger, A., Elst, G., Hanke, R. (eds.) Microelectronic Systems, pp. 243–252. Springer, Heidelberg (2011), http://dx.org/10.1007/978-3-642-23071-4_23
Scherer, S., Glodek, M., Schwenker, F., Campbell, N., Palm, G.: Spotting laughter in natural multiparty conversations: A comparison of automatic online and offline approaches using audiovisual data. ACM Trans. Interact. Intell. Syst. 2(1), 4:1–4:31 (2012)
Sundaram, S., Narayanan, S.: Automatic acoustic synthesis of human-like laughter. Journal of the Acoustical Society of America 121, 527–535 (2007)
Tokuda, K., Yoshimura, T., Masuko, T., Kobayashi, T., Kitamura, T.: Speech parameter generation algorithms for hmm-based speech synthesis. In: ICASSP, pp. 1315–1318 (2000)
Tokuda, K., Zen, H., Black, A.: An HMM-based speech synthesis system applied to English. In: Proceedings of the 2002 IEEE Speech Synthesis Workshop, Santa Monica, California, pp. 227–230 (2002)
Truong, K.P., van Leeuwen, D.A.: Automatic discrimination between laughter and speech. Speech Communication 49, 144–158 (2007)
Urbain, J., Çakmak, H., Dutoit, T.: Arousal-driven synthesis of laughter. Submitted to the IEEE Journal of Selected Topics in Signal Processing, Special Issue on Statistical Parametric Speech Synthesis (2014)
Urbain, J., Cakmak, H., Dutoit, T.: Development of hmm-based acoustic laughter synthesis. In: Interdisciplinary Workshop on Laughter and other Non-Verbal Vocalisations in Speech, Dublin, Ireland, pp. 26–27 (2012)
Urbain, J., Cakmak, H., Dutoit, T.: Evaluation of hmm-based laughter synthesis. In: International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Vancouver, Canada (2013)
Urbain, J., Dutoit, T.: A phonetic analysis of natural laughter, for use in automatic laughter processing systems. In: Proceedings of the Fourth Bi-annual International Conference of the HUMAINE Association on Affective Computing and Intelligent Interaction (ACII 2011), Memphis, Tennesse, pp. 397–406 (2011)
Urbain, J., Niewiadomski, R., Bevacqua, E., Dutoit, T., Moinet, A., Pelachaud, C., Picart, B., Tilmanne, J., Wagner, J.: Avlaughtercycle: Enabling a virtual agent to join in laughing with a conversational partner using a similarity-driven audiovisual laughter animation. Journal on Multimodal User Interfaces 4(1), 47–58 (2010); special Issue: eNTERFACE 2009
Urbain, J., Niewiadomski, R., Mancini, M., Griffin, H., Çakmak, H., Ach, L., Volpe, G.: Multimodal analysis of laughter for an interactive system. In: Mancas, M., d’ Alessandro, N., Siebert, X., Gosselin, B., Valderrama, C., Dutoit, T. (eds.) Intetain. LNICST, vol. 124, pp. 183–192. Springer, Heidelberg (2013)
Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., André, E.: The social signal interpretation (ssi) framework - multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM International Conference on Multimedia, Barcelona, Spain, October 21-25 (2013)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 IFIP International Federation for Information Processing
About this paper
Cite this paper
Mancini, M. et al. (2014). Laugh When You’re Winning. In: Rybarczyk, Y., Cardoso, T., Rosas, J., Camarinha-Matos, L.M. (eds) Innovative and Creative Developments in Multimodal Interaction Systems. eNTERFACE 2013. IFIP Advances in Information and Communication Technology, vol 425. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-55143-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-642-55143-7_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-55142-0
Online ISBN: 978-3-642-55143-7
eBook Packages: Computer ScienceComputer Science (R0)