Abstract
Often, designers and developers of serious games (and virtual simulations in general) strive for high fidelity (realism). However, real-time high-fidelity rendering of complex environments across sensory modalities such as vision, audition, and haptic (sense of touch) is still beyond our computational reach. Previous work has demonstrated that multimodal effects can be considerable, to the extent that a large amount of detail of one sense may be ignored or enhanced by the presence of other sensory inputs. Taking advantage of such multimodal effects, perceptual-based rendering—whereby the rendering parameters are adjusted based on the perceptual system—can be employed to limit computational processing. Motivated by the general lack of emphasis given to the understanding of audio rendering in virtual environments and games, we have started investigating multimodal (audiovisual) interactions within such virtual environments. Our work has shown that sound can directly affect visual fidelity perception and task performance within a virtual environment. These effects can be very individualized, whereby the influence of sound is dependent on various individual factors including musical listening preferences, suggesting the importance of individualizing the virtual environment to each user. This chapter begins with an overview of virtual environments and serious gaming’s open problems, with an emphasis on fidelity, and multimodal interactions, and the implications that these may have on performance and computational requirements. A detailed summary of our own prior work will be provided along with insight and suggestions that may guide designers and developers of serious games and virtual learning environments in general. Although the chapter is contextualized in the use of serious games in health professions education, the information provided is generalizable across a variety of domains.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
A complete discussion of the use of serious gaming and game-based technologies for health professions education is beyond the scope of this chapter. However, an overview of studies published focusing on serious games to teach some aspects of healthcare training is provided by de Ribaupierre et al. (2014) while an overview of serious gaming and virtual simulation and their application for surgical education and training is provided by Kapralos, Moussa, and Dubrowski (2014).
- 2.
Greater details regarding fidelity in general are available elsewhere. For example, Alexander, Brunyé, Sidman, and Weil (2005) provide a review of studies on fidelity, immersion, presence, and the resulting effects on transfer in simulations and games. Greater details regarding the influence of sound over visual rendering and task performance are provided by Hulusic et al. (2012) while an overview of multimodal influences on visual perception is provided by Shams and Kim (2010).
References
Alexander, A. L., Brunyé, T., Sidman, J., & Weil, S. A. (2005). From gaming to training: A review of studies on fidelity, immersion, presence, and buy-in and their effects on transfer in PC-based simulations and games. DARWARS Training Impact Group, November. Retrieved from CiteSeer.
Anyanwu, E. G. (2015). Background music in the dissection laboratory. Impact on stress associated with the dissection experience. Advances in Physiology Education, 39(2), 96–10.
Avila, C., Furnham, A., & McClelland, A. (2012). The influence of distracting familiar vocal music on cognitive performance of introverts and extraverts. Psychology of Music, 40(1), 84–93.
Becker, K., & Parker, J. (2012). The guide to computer simulations and games. Indianapolis, IN: Wiley.
Blascovich, J., & Bailenson, J. (2011). Infinite reality. New York, NY: Harper Collins.
Bonneel, N., Suied, C., Viaud-Delmon, I., & Drettakis, G. (2010). Bimodal perception of audio-visual material properties for virtual environments. ACM Transactions on Applied Perception, 7(1), 1–16.
Bracken, C., & Skalski, P. (2006, August). Presence and video games: The impact of image quality and skill level. Paper presented at the 9th Annual International Workshop on Presence. Retrieved from http://ispr.info/presence-conferences/previous-conferences/presence-2006/
Cannon-Bowers, J. (2006, March). The state of gaming and simulation. Paper presented at the Training 2006 Conference and Expo.
Chalmers, A., & Debattista, K. (2009, March). Levels of realism for serious games. Paper presented at the 2009 Conference in Games and Virtual Worlds for Serious Applications. doi:10.1109/VS-GAMES.2009.43.
Chandler, T., Anthony, M., & Klinger, D. (2009, June). Applying high cognitive vs, high physical fidelity within serious games. Paper presented at the Interservice/Industry Training, Simulation, and Education Conference.
Cheng, K., & Cairns, P. A. (2005, April). Behaviour, realism and immersion in games. Paper presented at the CHI ’05 Extended Abstracts on Human Factors in Computing Systems. doi:10.1145/1056808.1056894.
Cook, D. A., Hamstra, S. J., Brydges, R., Zendejas, B., Szostek, J. H., Wang, A. T., et al. (2013). Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis. Medical Teacher, 35(1), e867–e898.
Cooke, N. J., & Shope, S. M. (2004). Designing a synthetic task environment. In S. G. Schiflett, L. R. Elliott, E. Salas, & M. D. Coovert (Eds.), Scaled worlds: Development, validation, and application (pp. 263–278). Surry, England: Ashgate.
Cowan, B., Rojas, D., Kapralos, B., Collins, K., & Dubrowski, A. (2013, June). Spatial sound and its effect on visual quality perception and task performance within a virtual environment. Paper presented at the 21st International Congress on Acoustics. doi:http://dx.doi.org/10.1121/1.4798377.
Cowan, B., Rojas, D., Kapralos, B., Moussa, F., & Dubrowski, A. (2015). Effects of sound on visual realism perception and task performance. Visual Computer, 31(9), 1207–1216.
Cowan, B., Sabri, H., Kapralos, B., Porte, M., Backstein, D., Cristancho, S., et al. (2010). A serious game for total knee arthroplasty procedure education and training. Journal of Cybertherapy and Rehabilitation, 3(3), 285–298.
de Ribaupierre, S., Kapralos, B., Haji, F., Stroulia, E., Dubrowski, A., & Eagleson, R. (2014). Healthcare training enhancement through virtual reality and serious games. In M. Ma, C. Lakhmi, L. Jain, & P. Anderson (Eds.), Virtual, augmented reality and serious games for healthcare (pp. 9–27). Berlin, Germany: Springer.
Driskell, J. E., Johnston, J. H., & Salas, E. (2001). Does stress training generalize to novel settings? Human Factors, 43(1), 99–110.
Furnham, A., & Strbac, L. (2002). Music is as distracting as noise: The differential distraction of background music and noise on the cognitive test performance of introverts and extraverts. Ergonomics, 45(3), 203–217.
Godden, D. R., & Baddeley, A. D. (1975). Context dependent memory in two natural environments: On land and underwater. British Journal of Psychology, 66(3), 325–331.
Gopher, D. (2006). Emphasis change as a training protocol for high demand tasks. In A. Kramer, D. Wiegman, & A. Kirlik (Eds.), Attention: From theory to practice (pp. 209–224). New York, NY: Oxford Psychology Press.
Gosen, J., & Washbush, J. (2004). A review of scholarship on assessing experiential learning effectiveness. Simulation & Gaming, 35(2), 270–293.
Hays, R. T. (2005). The effectiveness of instructional games: A literature review and discussion (Technical Report 2005-004). Orlando, FL: Naval Air Warfare Center, Training Systems Division.
Hays, R. T., & Singer, M. (1989). Simulation fidelity in training system design. New York, NY: Springer.
Hulusic, V., Debattista, K., Aggarwal, V., & Chalmers, A. (2011). Maintaining frame rate perception in interactive environments by exploiting audio-visual cross-modal interaction. Visual Computer, 27(1), 57–66.
Hulusic, V., Harvey, C., Debattista, K., Tsingos, N., Walker, S., Howard, D., et al. (2012). Acoustic rendering and auditory-visual cross-modal perception and interaction. Computer Graphics Forum, 31(1), 102–131.
Isaranuwatchai, W., Brydges, R., Carnahan, H., Backstein, D., & Dubrowski, A. (2014). Comparing the cost-effectiveness of simulation modalities: A case study of peripheral intravenous catheterization training. Advances in Health Sciences Education, 19(2), 219–232.
Kapralos, B., Moussa, F., & Dubrowski, A. (2014). An overview of virtual simulations and serious games for surgical education and training. In M. Ma, C. Lakhmi, L. Jain, & P. Anderson (Eds.), Virtual, augmented reality and serious games for healthcare (pp. 289–306). Berlin, Germany: Springer.
Kapralos, B., Shewaga, R., & Ng, G. (2014). Serious games: Customizing the audio-visual interface. In R. Shumaker, & S. Lackey (Eds.), Virtual, augmented and mixed reality. Virtual, augmented and mixed reality. Applications of virtual and augmented reality. Lecture notes in computer science (Vol. 8526, pp. 190–199).
Ker, J., & Bradley, P. (2010). Simulation in medical education. In T. Swanwick (Ed.), Understanding medical education: Evidence, theory and practice (pp. 164–190). West Sussex, UK: Wiley-Blackwell.
Larsson, P., Västjäll, D., & Kleiner, M. (2003, April). On the quality of experience: A multi-modal approach to perceptual ego-motion and sensed presence in virtual environments. Paper presented at the First International Speech Communications Association Tutorial and Research Workshop on Auditory Quality of Systems. Retrieved from http://www.isca_speech.org/archive_open/aqs2003
Lee, C., Rincon, G. A., Meyer, G., Höllerer, T., & Bowman, D. A. (2013). The effects of visual realism on search tasks in mixed reality simulation. IEEE Transactions on Visual Computing and Graphics, 19(4), 547–556.
Mastoropoulou, G., Debattista, K., Chalmers, A., & Troscianco, T. (2005, August). The influence of sound effects on the perceived smoothness of rendered animations. Paper presented at the 2nd Symposium on Applied Perception in Graphics and Visualization. doi:10.1145/1080402.1080404.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychology, 38(1), 43–52.
McGaghie, W. C., Issenberg, S. B., Petrusa, E. R., & Scalese, R. J. (2010). A critical review of simulation-based medical education research: 2003–2009. Medical Educator, 44(1), 50–63.
McMahan, R. P. (2011). Exploring the effects of higher-fidelity display and interaction for virtual reality games. Doctoral dissertation. Retrieved from Digital Libraries and Archives (etd-12162011-140224).
Morris, C. S., Hancock, P. A., & Shirkey, E. C. (2004). Motivational effects of adding context relevant stress in PC-based game training. Military Psychology, 16(1), 135–147.
Muchinsky, P. M. (1989). Psychology applied to work. Summerfield, NC: Hypergraphic Press.
Netepezuk, D. W. (2013, July). Immersion and realism in video games—The confused moniker of video game engrossment. Paper presented at the 18th International Conference on Computer Games. doi:10.1109/CGames.2013.6632613.
Norman, G., Dore, K., & Grierson, L. (2012). The minimal relationship between simulation fidelity and transfer of learning. Medical Educator, 46(7), 636–647.
Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1–4.
Rademacher, P., Lengyel, J., Cutrell, E., & Whitted, T. (2001). Measuring the perception of visual realism in images. In S. J. Gortler, & K. Myszkowski (Eds.), Proceedings of the 12th Eurographics Workshop on Rendering Techniques (pp. 235–248): London, UK: Springer.
Rojas, D., Cowan, B., Kapralos, B., Collins, K., & Dubrowski, A. (2015, June). The effect of sound on visual quality perception and task completion time in a cel-shaded serious gaming virtual environment. Paper presented at the 7th IEEE International Workshop on Quality of Multimedia Experience. doi:10.1109/QoMEX.2015.7148136.
Rojas, D., Kapralos, B., Collins, K., & Dubrowski, A. (2014). The effect of contextual sound cues on visual fidelity perception. Studies in Health Technology and Informatics, 196, 346–352.
Rojas, D., Kapralos, B., Cristancho, S., Collins, K., Hogue, A., Conati, C., et al. (2012). Developing effective serious games: The effect of background sound on visual fidelity perception with varying texture resolution. Studies in Health Technology and Informatics, 173, 386–392.
Rojas, D., Kapralos, B., Crsitancho, S., Collins, K., Conati, C., & Dubrowski, A. (2011, September). The effect of background sound on visual fidelity perception. Paper presented at the ACM Audio Mostly 2011 Conference—6th Conference on Interaction with Sound. doi:10.1145/2095667.2095675.
Rojas, D., Kapralos, B., Hogue, A., Collins, K., Nacke, L., Crsitancho, C., et al. (2013). The effect of ambient auditory conditions on visual fidelity perception in stereoscopic 3D. IEEE Transactions on Systems, Man, and Cybernetics Part B: Cybernetics, 43(6), 1572–1583.
Seitz, A. R., van Wassenhove, V., & Shams, L. (2007). Simultaneous and independent acquisition of multisensory and unisensory associations. Perception, 36(10), 1445–1453.
Shams, L., & Kim, R. (2010). Crossmodal influences on visual perception. Physics Life Reviews, 7(3), 295–298.
Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning. In U. Ritterfeld, M. Cody, & P. Vorderer (Eds.), Serious games: Mechanisms and effects (pp. 295–321). New York, NY: Routedle.
Slater, M., Khanna, P., Mortensen, J., & Yu, I. (2009). Visual realism enhances realistic response in an immersive virtual environment. IEEE Computer Graphics and Applications, 29(3), 76–84.
Slater, M., & Wilbur, M. (1997). A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence Teleoperators and Virtual Environments, 6(6), 603–616.
Squire, K., & Jenkins, H. (2003). Harnessing the power of games in education. Insight, 3, 5–33.
Storms, S. L., & Zyda, M. J. (2000). Interactions in perceived quality of auditory-visual displays. Presence Teleoperators and Virtual Environments, 9(6), 557–580.
Sweller, J., Ayres, P. L., Kalyuga, S., & Chandler, P. A. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31.
Tashiro, J., & Dunlap, D. (2007, November). The impact of realism on learning engagement in educational games. Paper presented at the ACM 2007 Conference on Future Play. doi:10.1145/1328202.1328223.
Treisman, A. M., & Riley, J. G. (1969). Is selective attention selective perception or selective response? A further test. Journal of Experimental Psychology, 79(1), 27–34.
Veinott, E. S., Perleman, B., Polander, E., Leonard, J., Berry, G., Catrambone, R., et al. (2014, October). Is more information better? Examining the effects of visual and cognitive fidelity on learning in a serious video game. Paper presented at the 2014 IEEE Games Entertainment and Media Conference. doi:10.1109/GEM.2014.7048105.
Acknowledgments
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Social Sciences and Humanities Research Council of Canada (SSHRC), Interactive & Multi-Modal Experience Research Syndicate (IMMERSe) initiative, and the Canadian Network of Centres of Excellence (NCE), Graphics, Animation, and New Media (GRAND) initiative.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Kapralos, B., Moussa, F., Collins, K., Dubrowski, A. (2017). Fidelity and Multimodal Interactions. In: Wouters, P., van Oostendorp, H. (eds) Instructional Techniques to Facilitate Learning and Motivation of Serious Games. Advances in Game-Based Learning. Springer, Cham. https://doi.org/10.1007/978-3-319-39298-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-39298-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-39296-7
Online ISBN: 978-3-319-39298-1
eBook Packages: EducationEducation (R0)