Abstract
In real-world scenarios, humans estimate a large proportion of their perceived world contextually and use previous information to adjust or modify their expectations and responses. A typical example of congruence is when a smile (visual modality) is accompanied by a laugh (acoustic modality). Pain and pleasure are extremely intensive affects, rarely correctly assessed. It rarely happens that affective communication is incongruent in different modalities. Intentional combinations of these two affective expressions may be implemented by using audiovisual media. It is to be expected that presentations involving incongruent combinations during gaming and other interactions with machines will alter perceptions and the impact of ambiguity. To evaluate the impact of sensory crosstalk a novel statistical analysis was developed to estimate the impact of each modality. The results show that the visual modality dominates for the pairing pain/pleasure for. The acoustic modality dominates for pain/pleasure, pain/neutral, and pleasure/neutral. For neutral/pain and neutral/pleasure, neither the visual nor the acoustic modality enables highly correct ratings. The findings are of high value to the psychology of perception on a theoretical level and game/media developers as application fields.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bertelson, P, Aschersleben, G.: Temporal ventriloquism: crossmodal interaction on the time dimension. 1. Evidence from auditory-visual temporal order judgment. Int. J. Psychophysiol. 50(1–2), 147–155 (2003). https://doi.org/10.1016/S0167-8760(03)00130-2
Boschetti, S., Prossinger, H., Hladký, T., Machová, K., Binter, J.: “Eye can’t see the difference”: facial expressions of pain, pleasure, and fear are consistently rated due to chance. Hum. Ethol. 37, 46–72 (2022). https://doi.org/10.22330/he/37/046-072
Binter, J., Boschetti, S., Hladký, T., Prossinger, H.: “Ouch!” or “Aah!”: Are Vocalizations of ‘Laugh’, ‘Neutral’, ‘Fear’, ‘Pain’ or ‘Pleasure’ Reliably Rated? (2023). https://psyarxiv.com/rf7vw/
Brandner, J.L., Pohlman, J., Brase, G.L.: On hits and being hit on: error management theory, signal detection theory, and the male sexual overperception bias. Evol. Hum. Behav. 42(4), 331–342 (2021)
Caelen, O.: A Bayesian interpretation of the confusion matrix. Ann. Math. Artif. Intell. 81(3–4), 429–450 (2017). https://doi.org/10.1007/s10472-017-9564-8
Connor, S.: Dumbstruck: A Cultural History of Ventriloquism. Oxford University Press, Oxford (2000)
de Gelder, B., Pourtois, G., Weiskrantz, L.: Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures. Proc. Natl. Acad. Sci. USA 99(6), 4121–4126 (2002). https://doi.org/10.1073/pnas.062018499
Gregorić, B., et al.: Recognition of facial expressions in men and women. Medicina Fluminensis 50(4), 454–461 (2014)
Hess, U., Blairy, S., Kleck, R.E.: The intensity of emotional facial expressions and decoding accuracy. J. Nonverbal Behav. 21, 241–257 (1997)
Hutmacher, F.: Why is there so much more research on vision than on any other sensory modality? Front. Psychol. 10, 2246 (2019)
Kruschke, J.K.: Doing Bayesian Data Analysis. A Tutorial with R, JAGS, and STAN, 2nd edn. Elsevier/Wiley, Waltham (2015)
Perilloux, C., Easton, J.A., Buss, D.M.: The misperception of sexual interest. Psychol. Sci. 23(2), 146–151 (2012)
Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., Crommelinck, M.: The time-course of intermodal binding between seeing and hearing affective information. NeuroReport 11(6), 1329–1333 (2000). https://doi.org/10.1097/00001756-200004270-00036
Pourtois, G., de Gelder, B., Bol, A., Crommelinck, M.: Perception of facial expressions and voices and of their combination in the human brain. Cortex 41(1), 49–59 (2005). https://doi.org/10.1016/S0010-9452(08)70177-1
Shimojo, S., Shams, L.: Sensory modalities are not separate modalities: plasticity and interactions. Curr. Opin. Neurobiol. 11(4), 505–509 (2001). https://doi.org/10.1016/S0959-4388(00)00241-5
Vroomen, J., Driver, J., de Gelder, B.: Is cross-modal integration of emotional expressions independent of attentional resources? Cogn. Affect. Behav. Neurosci. 1(4), 382–387 (2001). https://doi.org/10.3758/CABN.1.4.382
Wada, Y., Kitagawa, N., Noguchi, K.: Audio-visual integration in temporal perception. Int. J. Psychophysiol. 50(1–2), 117–124 (2003). https://doi.org/10.1016/S0167-8760(03)00128-4
Funding and Acknowledgement
The research and data collection was supported by Czech Science Foundation (Grant No. 19-12885Y “Behavioral and Psycho-Physiological Response on Ambivalent Visual and Auditory Stimuli Presentation”).
Funding
The project was evaluated and approved by the Ethical Committee of the Faculty of Science, Charles University, as part of broader project (7/2018). GDPR regulations were followed at all times.
Conflicts of Interest.The authors declare no conflicts of interest.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Appendix
Appendix
We show a method of determining whether two sets of ratings are significantly different with an example (Fig. A-1). The rating entries of the female raters for the male stimuli are \({n}_{1}\) and the entries for the male raters for the male stimuli are \({n}_{2}\); then the Beta distribution is 
. The two boundaries \(\left[\frac{1}{2},{u}_{\mathrm{upper}}\right]\) of the HDI (Highest Density Interval) are determined by \(pdf\left(\frac{1}{2}\right)=pdf({u}_{\mathrm{upper}})\). (Comment: solving for \({u}_{\mathrm{upper}}\) requires computing power.) The probability of HDI is determined by
Comment: this integral can be easily computed using the CDF (Cumulative distribution function) of the Beta distribution: probability = 
If the computed probability is less than \(95\%\), then the significance level is greater than \(5\%\). In this case, the deviation of the mode from \(\frac{1}{2}\) is insignificant and the differences between the rating distributions are insignificant. In the example, shown in Fig. A-1, the probability is \(60\%\); the significance level is therefore 40% and the observed difference in the rating distributions is insignificant.
A graph showing the method of determining the significance of the difference between two ratings of the same modality by raters of different sex (female F versus male M) of the male stimuli m. In this example, the analysis for the acoustic modality is shown, and the rating numbers of female and male stimuli are the concentration parameters of the Beta distribution (each increased by + 1). \(s\) is the (Bayesian) probability. The mode of the distribution is shown, flanked by the upper and lower bounds of HDI60%. The dashed orange lines indicate equal likelihoods. The area shaded in orange is the probability that the mode is significant, in this case, \(100\%-60\%=40\%\). The mode is not significantly different from \(\frac{1}{2}\), and the ratings by the females and the males of the male stimuli is not significantly different at the 5% significance level.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Boschetti, S., Prossinger, H., Prossinger-Beck, V., Hladký, T., Říha, D., Binter, J. (2023). Never Correct: The Novel Analysis of Differing Visual (Facial Expression) and Acoustic (Vocalization) Bimodal Displays of the Affective States “Pain”, “Pleasure”, and “Neutral”. In: Fang, X. (eds) HCI in Games. HCII 2023. Lecture Notes in Computer Science, vol 14047. Springer, Cham. https://doi.org/10.1007/978-3-031-35979-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-35979-8_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-35978-1
Online ISBN: 978-3-031-35979-8
eBook Packages: Computer ScienceComputer Science (R0)