Multitouch

An increasing number of today's consumer devices such as mobile phones or tablet computers are equipped with various sensors. The extraction of useful information such as gestures from sensor-generated data based on mainstream imperative languages is a notoriously difficult task. Over the last few years, a number of domain-specific programming languages have been proposed to ease the development of gesture detection. Most of these languages have adopted a declarative approach allowing programmers to describe their gestures rather than having to manually maintain a history of event data and intermediate gesture results. While these declarative languages represent a clear advancement in gesture detection, a number of issues are still unresolved. In this paper we present relevant criteria for gesture detection and provide an initial classification of existing solutions based on these criteria in order to foster a discussion and identify opportunities for future gesture programming languages.

Saggio su interazione uomo macchina, usabilità, valore euristico dell'arte, le più diffuse tecnologie d'interfaccia, mems, gesture recognition.
Contiene i seguenti case studies: smart laser scanner, physical motion interface, sensorizzazione di una protesi, Apple multitouch screen, Nintendo WiiMote.

In particolare, al momento della pubblicazione nell'Aprile 2006, il saggio conteneva alcuni indizi di retro-ingegneria sul controller Wii, procurato tramite l'Università quando ancora non era disponibile al pubblico (nei negozi a Dicembre 2006).

Multi-touch interaction with computationally enhanced surfaces has received considerable recent attention. Approaches to the implementation of multi-touch interaction such as Frustrated Total Internal Reflection (FTIR) and Diffused Illumination (DI) have allowed for the low cost development of such surfaces, leading to a number of technology and application innovations. Although many of these techniques have been presented in an academic setting, the practicali-ties of building a high quality multi-touch enabled surface, both in terms of the software and hardware, are not trivial. This document aims to summarize the knowledge and experience of developers of multi-touch technology who gathered at the Bootcamp on Construction & Implementation of Optical Multi-touch Surfaces at Tabletop 2008 in Ams-terdam, and seeks to provide hints and practical advice to people seeking to " build your own " multi-touch surface. We mostly focus on technical aspects that are important in the construction of optical multi-touch surfaces, including: infrared illumination, silicone compliant surfaces, projection screens, cameras, filters, and projectors. In addition, we outline how to integrate this hardware to allow users to create a solid multi-touch surface, and provide an overview of existing software libraries for the implementation of multi-touch applications. In addition, we discuss the problem of latency introduced by the different parts of the system. A brief description of most of the common technologies to realize (multi-) touch surfaces is provided; however, the main focus is on those that utilise optical approaches.

The information society manufactures, manipulates, and commodifies information. Heritage is one such area that is undergoing digital transformation. Heritage is increasingly being transmuted through digitisation devices such as laser and structured light scans into multiple representations of information. The rich information of a heritage object or an environment can be restructured, transmitted, and recomposed into a mediated form both textual and non-textual. Once digitised, it becomes free from its physical predecessor; it enters another world that defies the physical laws of nature where the imagination of the maker is a limit. Such worlds accompanied by their objects are accessible in new yet intuitive ways via surface computers. The horizontal nature of the multitouch-multiuser surface computer then becomes the mirror that links both worlds, allowing access into a virtual space via the touch-table computing paradigm. This article explores 3D surface computing, its technology, capabilities and limits with developments of two multitouch applications incorporating heritage objects and environments, and the observation of the reactions of initial users. It addresses new issues and challenges surrounding the use of surface computing and how the access and transmission of heritage information via multitouch-multiuser tables are able to contribute to the accessibility, teaching and learning of heritage.

"Analogous perceptual experiences in multiple sensory modalities provide possibilities for abstract expression using modern computer-based audio-visual systems. This work examines the phenomenon of such cross-modal links with a focus on mappings between the auditory and visual realms.
The design and implementation of a graphics-based sound synthesis controller using such inter-sensory associations is presented. A review of literature relevant to the design of computer-based musical instruments is provided, including discussions of parameter mapping, the use of graphical displays and computer vision as a gesture capture mechanism.
An analysis of the software instruments realised and the physical interface setup is provided. System improvements and possible applications are also discussed with the direction of future work with the system being suggested."

Although multi-touch applications and user interfaces have become increasingly common in the last few years, there is no agreed-upon multi-touch user interface language yet. In order to gain a deeper understanding of the design of multi-touch user interfaces, this paper presents semiotic analysis of multi-touch applications as an interesting approach to gain deeper understanding of the way users use and understand multi-touch interfaces. In a case study example, user tests of a multi-touch tabletop application platform called MuTable are analysed with the Communicability Evaluation Method to evaluate to what extent users understand the intended messages (e.g., cues about interaction and functionality) the MuTable platform communicates. The semiotic analysis of this case study shows that although multi-touch interfaces can facilitate user exploration, the lack of well-known standards in multi-touch interface design and in the use of gestures makes the user interface difficult to use and interpret. This conclusion points to the importance of the elusive balance between letting users explore multi-touch systems on their own on one hand, and guiding users, explaining how to use and interpret the user interface, on the other.► We present semiotic analysis as a useful approach to study multi-touch interaction. ► A tabletop user interface is studied from the perspective of Semiotic Engineering. ► We propose design characteristics for a multi-touch user guidance system.

In this paper we discuss our experiences of conducting user evaluations of technological installations for the new Robert Burns Birthplace museum and the challenges associating with doing this. We believed it to be important to avoid a sterile lab situation, and to be able to observe the social dynamics that evolve around museum installations. As we were unable to conduct studies in the museum itself we devised ways of taking them out of the lab by basing them in a more museum-like or semi-realistic setting. We took efforts to recreate the installation setup when this was likely to influence interaction patterns and could reveal usability issues. Moreover, we invited young families and adult groups to take part in the evaluation sessions to investigate how they would interact with these as a group. We explain our motivations and the tradeoffs involved. We focus on discussing the effects that the semi-realistic setting may have had on the user feedback obtained, reflect on the differing user behaviours that might emerge in the genuine museum setting and make recommendations for future studies.

3D modeling has been revolutionized in recent years by the
advent of computers. While computers have become much more affordable and accessible to the masses, computer modeling remains  a complex task involving  a  steep learning curve and extensive training.  In this paper we describe  the MozArt Table, our effort to redefine the interface for computer modeling to make  it more accessible to lay users. We have explored both the  hardware and software aspects  of the interface, specifically, the use of intuitive speech commands and multitouch gestures on an inclined interactive surface.  The paper describes our approach,
hardware setup and the technology used to make it work.

Abstract This research examines phonics instruction on multi-touch display technology for collaborative learning. The paper provides a brief discussion of technology, collaborative learning and literacy instruction and then focuses on the design of a Phonics application using collaborative learning interactions. The paper discusses the methodology of a multi-phase evaluation of the usability and the instructional design of the application. A survey was used for the usability study and a focus group consisting of experts in the field of ...

MorphOSC is a new toolkit for building graphical user interfaces for the control of sound using morphing between parameter presets. It uses the multidimensional interpolation space paradigm seen in some other systems, but hitherto unavailable as open-source software in the form presented here. The software is delivered as a class library for the Processing Development Environment and is cross-platform for desktop
computers and Android mobile devices. This paper positions the new library within the context of similar software, introduces the main features of the initial code release and details future work on the project.

In this paper we present the table-based tangible interface application Xenakis which uses probability models in order to compose music in a way that can be strongly influenced by the user. Our musical sequencing application is based on a framework for tangible interfaces with an architecture that is strongly inspired by the model-view-controller pattern. In addition, we developed a hardware setup for tangible interfaces and used MatraX for tracking markers. The sequencer is the first implementation based on this framework. It allows users to create music simply by moving tangibles on the table. The graphics engine Horde3D is used to visualize the user-interaction and to show the relationships between the tangible objects on the table, creating an appealing audio-visual experience. An evaluation with 37 first time users was conducted in order to discover the strong and the weak points of such tangible user interfaces, especially in the context of our application.

Two technology trends address the rising costs of healthcare systems in aging societies: Serious Games for Healthcare and Ambient Assisted Living Environments. Surprisingly, these concepts are rarely combined and the users’ perception and use of Serious Games in Ambient Assisted Living environments is insufficiently understood. We present the evaluation of a serious game for stimulating cognitive abilities for elderly with regard to technology acceptance (based on the UTAUT2 model), performance and preference for an interaction device (tablet, table, wall). The results suggest that acceptance of serious games is independent of gender, technical expertise, gaming habits, and only weakly influenced by age. Determinants for acceptance are perceived fun and the feeling that the users can make playing the game a habit. Performance within the game is explained by age and previous gaming experience. All investigated interaction devices were rated as useful and easy to learn, although the wall-sized display had lower approval levels. The article concludes with guidelines for successfully introducing serious games for healthcare to residents in ambient assisted living environments.

This research applies Socio-Cultural theory and Distributed Cognition/Activity theory to conceptualize the design of collaborative learning activities in large shared digital spaces. The paper begins by providing a summary of previous work in the creation of a technology platform for large shared digital spaces. It then details how Socio-Cultural theory and Distributed Cognition/Activity theory can provide significant insights into the educational utility of such collaborative technology and how design issues relating to collaborative learning may be addressed. The paper then reports on an experimental study involving a Phonics literacy application, including data from a survey of 19 participants and focus groups sessions by 6 discipline experts. Themes related to Socio-Cultural theory and Distributed Cognition/Activity theory are highlighted in the results reported. Conclusions and further research opportunities involving large shared digital learning spaces are then discussed.

Over the past few years, multi-touch user interfaces emerged from research prototypes into mass market products. This evolution has been mainly driven by innovative devices such as Apple's iPhone or Microsoft's Surface tabletop computer. Unfortunately, there seems to be a lack of software engineering abstractions in existing multi-touch development frameworks. Many multi-touch applications are based on hard-coded procedural low level event processing. This leads to proprietary solutions with a lack of gesture extensibility and cross-application reusability. We present Midas, a declarative model for the definition and detection of multi-touch gestures where gestures are expressed via logical rules over a set of input facts. We highlight how our rule-based language approach leads to improvements in gesture extensibility and reusability. Last but not least, we introduce JMidas, an instantiation of Midas for the Java programming language and describe how JMidas has been applied to implement a number of innovative multi-touch gestures.

The penetration of stereoscopic 3D (S3D) output devices is becoming widespread. S3D screens range in size from large cinema screens, to tabletop displays, TVs in living rooms, and even mobile devices are nowadays equipped with autostereoscopic 3D screens. As a consequence, the requirement for interacting with 3D content is also increasing, with " touch " being one of the dominant input methods. However, the design requirements and best practices for interaction with S3D touch screen user interfaces are only now evolving. To understand the challenges and limitations S3D technology brings to interaction design, and in particular the additional demands they place on the users of such interfaces, we present our research that considers stereoscopic touch screens of different sizes in static and mobile usage contexts. We report on perceptual, cognitive and behavioral aspects revealed by user studies and present interaction design requirements based on the findings.

Existing interaction paradigms are associated with direct manipulation with indirect input devices such as the mouse.
By accounting the emergency of multitouch interactivity on reactive surfaces, it becomes relevant to discuss upon the changes brought by the direct relation of the body, trough the hand, with the contents and objects represented on surface.
From the study of the scientific literature of the elements of the interaction and the principal gestural interaction techniques, we propose a methodological framework for the design gestural interaction based on three main approaches: empirical, metaphorical and phenomenological.

—We are motivated to find a multi-touch gesture detection algorithm that is efficient, easy to implement, and prospectively scalable to real-time applications using 3D environments. Our approach tries to solve the recognition for gestures with the use of feature extraction without the need of any previous learning samples. Before showing our proposed solution, we describe some algorithms that attempt to solve similar problems. Finally, we describe our code to accomplish off-line gesture recognition.

Physical computing empowers people to design and customize electronic hardware tailored to their individual needs. This often involves ”tinkering” with components connections, but due to the intangible nature of electricity, this can be difficult, especially for novices. We use a multi-stage design process to design, build and evaluate a physical prototyping workspace for novices to learn about real physical computing hardware. The workspace consists of a horizontal surface that tracks physical components like sensors, actuators, and microcontroller boards and augments them with additional digital information in situ. By digitally exploring various means of connecting components, users can experientially learn how to build a functioning circuit and then transition directly to building it physically. In a user study, we found that this system motivates learners by encouraging them and building a sense of competence, while also providing a stimulating experience.

Abstract Natural user interfaces (NUI) such as multitouch and surface computing are positioned as the next major evolution in computing and user interfaces. Just graphical user interfaces (GUIs) brought unprecedented interaction capabilities to their command-line predecessors, we believe multitouch and surface computing will spawn novel ways to interact with media and improve social usage patterns. Since experimentation and deployment are currently limited, the exploration of applications and interfaces in this area ...

We present the Rizzo, a multi-touch virtual mouse that has been designed to provide the fine grained interaction for information visualization on a multi-touch table. Our solution enables touch interaction for existing mouse-based visualizations. Previously, this transition to a multi-touch environment was difficult because the mouse emulation of touch surfaces is often insufficient to provide full information visualization functionality. We present a unified design, combining many Rizzos that have been designed not only to provide mouse capabilities but also to act as zoomable lenses that make precise information access feasible. The Rizzos and the information visualizations all exist within a touch-enabled 3D window management system. Our approach permits touch interaction with both the 3D windowing environment as well as with the contents of the individual windows contained therein. We describe an implementation of our technique that augments the VisLink 3D visualization environment to demonstrate how to enable multi-touch capabilities on all visualizations written with the popular prefuse visualization toolkit.

Most tabletop research presents findings from lab-based user studies, focusing on specific interaction techniques. This means we still know little about how these new interfaces perform in real life settings and how users appropriate them.

This paper presents findings from a field study of an existing interactive table in a museum of natural history. Visitors were found to employ a wide variety of gestures for interacting; different interface elements invited different types of gesture. The analysis highlights challenges and design conflicts in the design of tabletop interfaces for public settings, such as latency times and side-effects of ‘frame-less’ content, which had some users struggling to learn how to interact. While the majority of visitors engaged at least briefly with the table, which enabled browsing question-answer text about animal species, talk amongst visitors dealt mainly with how to interact and evoked few comments, indicating shallow
engagement with content.

Designing and assessing systems to support neighborhood participation in design is difficult due to the challenges of involving real participants and the fragile nature of early instantiations of technologies aimed at supporting open-ended and ill-structured design tasks. We report on a scenario-based, semi-realistic field trial of two prototypes of the Envisionment and Discovery Collaboratory, an environment for supporting community involvement in design activities. By engaging subjects in playing participant roles, we have been able to gain some crucial insights into the facets of the design at multiple levels as part of an ongoing design process.