TSpace

Abstract (summary): Think-aloud protocols are a highly valued usability testing method for identifying usability problems. Despite the value of conducting think-aloud usability test sessions, analyzing think-aloud sessions is often arduous. Consequently, previous research has urged the community to develop methods to support fast-paced analysis. Inspired by the research that shows subtle patterns in how we interact with other people reveal our attitudes toward them, I study subtle patterns in users’ verbalizations and speech when they encounter problems in think-aloud sessions and further leverage these patterns to support the analysis of think-aloud sessions. In this dissertation, I first survey user experience (UX) practitioners around the world to understand their practices and challenges around using think-aloud protocols. I then design and conduct three studies, each addressing the limitations of the previous one, to identify and validate the subtle patterns in users’ verbalization and speech features that tend to occur when they encounter usability problems. Informed by the findings from the studies, I take the first step to designing computational methods that leverage these subtle patterns and the power of machine learning (ML) to detect the usability problem encounters. To help UX practitioners leverage ML-inferred usability problem encounters, I design and evaluate an intelligent visual analytics tool that present UX practitioners with a timeline visualization of ML-inferred problem encounters and ML’s input features among other functions. Experimental results demonstrate that ML-inferred problem encounters help UX practitioners consider problems that they might have overlooked and therefore identify more usability problems. Moreover, I offer insights into how UX practitioners leverage and perceive ML-inferred problem encounters and ML’s input features in their analysis and their session review strategies (i.e., how they play, pause, rewind the recorded sessions). Finally, I highlight the promising directions to further forge a better symbiosis relationship between UX practitioners and machine intelligence when analyzing recorded think-aloud sessions.