research-article

Measuring user experience in automated driving: Developing a single-item measure

Authors:

Chantal Himmels,

Oliver Jarosch,

Axel BuchnerAuthors Info & Claims

AutomotiveUI '21: 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

Pages 191 - 200

https://doi.org/10.1145/3409118.3475135

Published: 20 September 2021 Publication History

Abstract

Measuring user experience is highly important for human-centered development and thus for designing automated driving systems. Multi-item measures such as the System Usability Scale (SUS) [7] or the Usability Metric for User Experience (UMUX) [14] are commonly used for collecting user feedback on technical systems or products. The goal of the present study was to investigate the potentials of a single-item approach as an economic alternative for measuring user experience compared to multi-item scales. Therefore, a single-item measure was developed to assess both event-related and cumulative user experience in automated driving. User experience was manipulated in a between-subject design implemented in a real-world driving task and feedback was collected using the newly developed Single Item User Experience (SIUX) scale, the UMUX, and the SUS. Results indicate that the SIUX scale is more sensitive than the UMUX to differences in event-related user experience, but not in cumulative user experience. Both the SIUX and the UMUX were more sensitive than the SUS when measuring differences in cumulative user experience. Future studies should be aimed at investigating the applicability of the SIUX scale to domains other than automated driving and at collecting more extensive data on validity and reliability of all three instruments.

References

[1]

Aaron Bangor, Phil Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies 4, 3, 114–123.

Digital Library

[2]

Klaus Bengler, Kamil Omozik, and Andrea I. Müller. 2020. The renaissance of Wizard of Oz (WoOz) - using the WoOz methodology to prototype automated vehicles. Proceedings of the Human Factors and Ergonomics Society Europe Chapter 2019 Annual Conference, 63–72.

[3]

Mehmet I. Berkman and Dilek Karahoca. 2016. Re-assessing the Usability Metric for User Experience (UMUX) scale. Journal of Usability Studies 11, 3, 89–109.

Digital Library

[4]

Nigel Bevan. 2008. Classifying and selecting ux and usability measures. In Proceedings of the International Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM). Institute of Research in Informatics of Toulouse, Toulouse, France, 13–18.

[5]

Simone Borsci, Stefano Federici, Silvia Bacci, Michela Gnaldi, and Francesco Bartolucci. 2015. Assessing user satisfaction in the era of user experience: Comparison of the SUS, UMUX and UMUX - LITE as a function of product experience. International Journal of Human-Computer Interaction 31, 8, 484–495.

[6]

George E. P. Box. 1953. Non-normality and tests on variance. Biometrika 40, 3/4, 318–335.

[7]

John Brooke. 1996. SUS - A quick and dirty usability scale. Usability evaluation in industry 194, 189.

[8]

Heiner Bubb, Klaus Bengler, Rainer E. Grünen, and Mark Vollrath. 2015. Automobilergonomie. Springer, Berlin, Germany.

[9]

Gilbert A. Churchill. 1979. A paradigm for developing better measures of marketing constructs. Journal of Marketing Research 16, 1, 64–73.

[10]

Eli P. Cox. 1980. The optimal number of response alternatives for a scale: A review. Journal of Marketing Research 17, 4, 407–422.

[11]

Lee J. Cronbach and Paul E. Meehl. 1955. Construct validity in psychological tests. Psychological bulletin 52, 4, 281–302.

[12]

Don A. Dillman, Michael D. Sinclair, and Jon R. Clark. 1993. Effects of questionnaire length, respondent-friendly design, and a difficult question on response rates for occupant-addressed census mail surveys. Public Opinion Quarterly 57, 3, 289–304.

[13]

Aimee L. Drolet and Donald G. Morrison. 2001. Do we really need multiple-item measures in service research? Journal of Service Research 3, 3, 196–204.

[14]

Kraig Finstad. 2010. The usability metric for user experience. Interacting with Computers 22, 5, 323–327.

Digital Library

[15]

Yannick Forster, Frederik Naujoks, Alexandra Neukum, and Lynn Huestegge. 2017. Driver compliance to take-over requests with different auditory outputs in conditional automation. Accident Analysis and Prevention 109, 18–28.

[16]

Christoph Fuchs and Adamantios Diamantopoulos. 2009. Using single-item measures for construct measurement in management research: Conceptual issues and application guidelines. Die Betriebswirtschaft 69, 2, 195–210.

[17]

Gene V. Glass, Percy D. Peckham, and James R. Sanders. 1972. Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. Review of Educational Research 42, 3, 237–288.

[18]

Rex Hartson and Pardha S. Pyla. 2012. The UX Book. Process and guidelines for ensuring a quality user experience. Elsevier, Amsterdam, Netherlands.

Digital Library

[19]

Sebastian Hergeth. 2016. Automation trust in conditional automated driving systems: Approaches to operationalizations and designs. PhD Thesis. Technische Universität Chemnitz, Chemnitz, Germany.

[20]

ISO 9241-11. 2018. Ergonomics of human-system interaction. Part 11: Usability: Definitions and concepts (2018). Retrieved March 21, 2021 from https://www.iso.org/obp/ui/.

[21]

Patricia S. Jones, Jerry W. Lee, Linda R. Phillips, Xinwei E. Zhang, and Karen B. Jaceldo. 2001. An adaptation of Brislin's translation model for cross-cultural research. Nursing research 50, 5, 300–304.

[22]

Daniel Kahneman and Jason Riis. 2005. Living, and thinking about it: two perspectives on life. In The science of well-being, Felicia A. Huppert, Nick Baylis and Barry Keverne, Eds. Oxford University Press, Oxford, England, 284–305.

[23]

Evangelos Karapanos, Marc Hassenzahl, and Jean B. Martens. 2008. User experience over time. In Proceeding of the twenty-sixth annual CHI conference extended abstracts on Human factors in computing systems - CHI '08. ACM, New York, NY, 3561–3566.

[24]

Moritz Körber and Klaus Bengler. 2013. Measurement of momentary user experience in an automotive context. In AutomotiveUI '13: Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, New York, NY, 194–201.

Digital Library

[25]

James R. Lewis. 2018. The System Usability Scale: Past, present, and future. International Journal of Human-Computer Interaction 34, 7, 577–590.

[26]

James R. Lewis, Brian S. Utesch, and Deborah E. Maher. 2015. Investigating the correspondence between UMUX-LITE and SUS scores. In Design, user experience, and usability: Design discourse, Aaron Marcus, Ed. Lecture Notes in Computer Science. Springer International Publishing, Cham, Switzerland, 204–211.

[27]

Andrea I. Müller, Veronika Weinbeer, and Klaus Bengler. 2019. Using the wizard of Oz paradigm to prototype automated vehicles. In AutomotiveUI '19: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, New York, NY, 181–186.

Digital Library

[28]

Mark S. Nagy. 2002. Using a single-item approach to measure facet job satisfaction. Journal of Occupational and Organizational Psychology 75, 1, 77–86.

[29]

Donald A. Norman and Stephen W. Draper, Eds. 1986. User centered system design. New perspectives on human-computer interaction. CRC Press, Boca Raton, London, New York.

[30]

Ralph G. O'Brien and Mary K. Kaiser. 1985. MANOVA method for analyzing repeated measures designs: An extensive primer. Psychological bulletin 97, 2, 316–333.

[31]

Kamil Omozik. In press. Expertenbewertungen als Absicherungsmethodik zum Nutzererlebnis beim automatisierten Fahren. PhD Thesis. Technische Universität München, München.

[32]

Joanne C. Rogan and H. J. Keselman. 1977. Is the ANOVA F-Test robust to variance heterogeneity when sample sizes are equal? An investigation via a coefficient of variation. American Educational Research Journal 14, 4, 493–498.

[33]

Bernd Rohrmann. 1978. Empirische Studien zur Entwicklung von Antwortskalen für die sozialwissenschaftliche Forschung. Zeitschrift für Sozialpsychologie 9, 3, 222–245.

[34]

SAE International. 2018. Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems (2018). Retrieved March 21, 2021 from https://www.sae.org/standards/content/j3016_201806/.

[35]

Marko Sarstedt and Petra Wilczynski. 2009. More for less? A comparison of single-item and multi-item measures. Die Betriebswirtschaft 69, 2, 211–228.

[36]

Vida Scarpello and John Campbell. 1983. Job satisfaction: Are all the parts there? Personnel Psychology 36, 3, 577–600.

[37]

Daniela Schlütz and Helmut Scherer. 2001. Der Einsatz der Experience Sampling Method in der Medienwissenschaft. Zeitschrift für Medienpsychologie 13, 3, 146–149.

[38]

Boas Shamir and Ronit Kark. 2004. A single-item graphic scale for the measurement of organizational identification. Journal of Occupational and Organizational Psychology 77, 1, 115–123.

[39]

John P. Wanous and Michael J. Hudy. 2001. Single-item reliability: A replication and extension. Organizational Research Methods 4, 4, 361–375.

[40]

John P. Wanous, Arnon E. Reichers, and Michael J. Hudy. 1997. Overall job satisfaction: how good are single-item measures? The Journal of applied psychology 82, 2, 247–252.

[41]

Mohammad Zarour and Mubarak Alharbi. 2017. User experience framework that combines aspects, dimensions, and measurement methods. Cogent Engineering 4, 1, 1421006.

Cited By

Gräve EBuchner A(2024)Is Less Sometimes More? An Experimental Comparison of Four Measures of Perceived UsabilityHuman Factors: The Journal of the Human Factors and Ergonomics Society10.1177/0018720824123786267:1(32-48)Online publication date: 14-Mar-2024
https://doi.org/10.1177/00187208241237862
Gräve EBell RBuchner A(2024)Verbal and pictorial single-item scales are as good as their 10-item counterparts for measuring perceived usabilityErgonomics10.1080/00140139.2024.237106167:12(2096-2111)Online publication date: 28-Jun-2024
https://doi.org/10.1080/00140139.2024.2371061
Buchner CKraus JMiller LBaumann M(2024)What is good? Exploring the applicability of a one item measure as a proxy for measuring acceptance in driver-vehicle interaction studiesJournal on Multimodal User Interfaces10.1007/s12193-024-00432-1Online publication date: 16-Jul-2024
https://doi.org/10.1007/s12193-024-00432-1

Recommendations

From Manual Driving to Automated Driving: A Review of 10 Years of AutoUI
AutomotiveUI '19: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

This paper gives an overview of the ten-year development of the papers presented at the International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutoUI) from 2009 to 2018. We categorize the topics into two main ...
A Comparison of SUS, UMUX-LITE, and UEQ-S

A loyal customer base depends upon a good user experience over the product's complete lifetime. Successful products are continuously developed over a long period. Their functionality and complexity typically grow over years, so it is important to ...
Optimizing User Interaction with MABIS: An Examination of Usability and User Experience in the Mobile Automated Biometric Identification System
ICCTA '24: Proceedings of the 2024 10th International Conference on Computer Technology Applications

This study focuses on the evaluation of the Mobile Automated Biometric Identification System (MABIS) through a dual approach incorporating both quantitative and qualitative methodologies. Utilizing the System Usability Scale (SUS) for gauging usability ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AutomotiveUI '21: 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

September 2021

306 pages

ISBN:9781450380638

DOI:10.1145/3409118

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 September 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

AutomotiveUI '21

Sponsor:

SIGCHI

AutomotiveUI '21: 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

September 9 - 14, 2021

Leeds, United Kingdom

Acceptance Rates

Overall Acceptance Rate 248 of 566 submissions, 44%

Upcoming Conference

AutomotiveUI '25

Sponsor:
sigchi

17th International Conference on Automotive User Interfaces and Interactive Vehicular Applications

September 21 - 25, 2025

Brisbane , QLD , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
464
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)7

Reflects downloads up to 20 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gräve EBuchner A(2024)Is Less Sometimes More? An Experimental Comparison of Four Measures of Perceived UsabilityHuman Factors: The Journal of the Human Factors and Ergonomics Society10.1177/0018720824123786267:1(32-48)Online publication date: 14-Mar-2024
https://doi.org/10.1177/00187208241237862
Gräve EBell RBuchner A(2024)Verbal and pictorial single-item scales are as good as their 10-item counterparts for measuring perceived usabilityErgonomics10.1080/00140139.2024.237106167:12(2096-2111)Online publication date: 28-Jun-2024
https://doi.org/10.1080/00140139.2024.2371061
Buchner CKraus JMiller LBaumann M(2024)What is good? Exploring the applicability of a one item measure as a proxy for measuring acceptance in driver-vehicle interaction studiesJournal on Multimodal User Interfaces10.1007/s12193-024-00432-1Online publication date: 16-Jul-2024
https://doi.org/10.1007/s12193-024-00432-1

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten