This article presents design and performance practices for movement-based digital musical instrum... more This article presents design and performance practices for movement-based digital musical instruments. We develop the notion of borrowed gestures, which is a gesture-first approach that composes a gestural vocabulary of nonmusical body movements combined with nuanced instrumental gestures. These practices explore new affordances for physical interaction by transferring the expressive qualities and communicative aspects of body movements; these body movements and their qualities are borrowed from nonmusical domains. By merging musical and nonmusical domains through movement interaction, borrowed gestures offer shared performance spaces and cross-disciplinary practices.Our approach centers on use of the body and the design with body movement when developing digital musical instruments. The performer's body becomes an intermediate medium, physically connecting and uniting the performer and the instrument. This approach creates new ways of conceptualizing and designing movement-base...
Inclusive musical instruments benefit from incorporating wearable interfaces into digital musical... more Inclusive musical instruments benefit from incorporating wearable interfaces into digital musical instrument design, creating opportunities for bodily felt experiences and movement-based interactions. In this article, we discuss the evolution of our inclusive design approach behind the design and performance practices of three wearable musical instruments. By focusing on the embodied, somatic, and tacit dimensions of movement-based musical interaction, we evaluate these case studies, combining the third and first-person perspectives. The design and implementation of the wearable sensing, utilizing the additive manufacturing techniques, are discussed for each instrument and its performer in specific cases of musical expression. This article further discusses how our approach integrates music performance as a crucial step into design and evaluation, utilizing these performance practices and such collaborative settings for improved diversity and inclusion. Finally, we examine how our design approach evolves from user-centered design to more participatory practices, offering people with diverse abilities a shared music performance space.
Inclusive musical instruments benefit from incorporating wearable interfaces into digital musical... more Inclusive musical instruments benefit from incorporating wearable interfaces into digital musical instrument design, creating opportunities for bodily felt experiences and movement-based interactions. In this article, we discuss the evolution of our inclusive design approach behind the design and performance practices of three wearable musical instruments. By focusing on the embodied, somatic, and tacit dimensions of movement-based musical interaction, we evaluate these case studies, combining the third and first-person perspectives. The design and implementation of the wearable sensing, utilizing the additive manufacturing techniques, are discussed for each instrument and its performer in specific cases of musical expression. This article further discusses how our approach integrates music performance as a crucial step into design and evaluation, utilizing these performance practices and such collaborative settings for improved diversity and inclusion. Finally, we examine how our d...
Cochlear implant (CI) users experience several challenges when listening to music. However, their... more Cochlear implant (CI) users experience several challenges when listening to music. However, their hearing abilities are greatly diverse and their musical experiences may significantly vary from each other. In this research, we investigate this diversity in CI users' musical experience, preferences, and practices. We integrate multisensory feedback into their listening experiences to support the perception of specific musical features and elements. Three installations are implemented, each exploring different sensory modalities assisting or supporting CI users' listening experience. We study these installations throughout semi-structured and exploratory workshops with participants. We report the results of our process-oriented assessment of CI users' experience with music. Because the CI community is a minority participant group in music, musical instrument design frameworks and practices vary from those of hearing cultures. We share guidelines for designing multisensory integration that derived from our studies with individual CI users and specifically aimed to enrich their experiences.
The fast-growing technology of additive manufacturing(AM), 3D printing, enables unique features i... more The fast-growing technology of additive manufacturing(AM), 3D printing, enables unique features in numerous design areas including musical instrument design. The recent advancements further introduce incorporating electronics into 3D printing. While still in the proof of concept phase, the ability to fabricate embedded sensors with 3D printing offers promising applications for musical instrument design. In this paper, embedding sensors and functional parts in the additive manufacturing process is discussed. We present the design process of embedded sensing for rapid prototyping digital musical instruments (DMI). The establishment of 3D models and operation sequences for different musical applications can be challenging since they vary depending on the electronic components. In order to address this problem, we provide a set of design and manufacturing specifications to customize the prototyping process for embedding structural electronics. The application of this hybrid method is de...
Vibrato Dataset is the extended version of the previous dataset. It consists of vibrato sounds fo... more Vibrato Dataset is the extended version of the previous dataset. It consists of vibrato sounds for five different instruments providing different pitch ranges of notes, melodies, articulations, scales and musical techniques as staccato and legato. It includes 295 note tracks with vibratos and corresponding non vibrato tones and 49 melody and articulation tracks. Data Collection It is organized according to the source of the recordings. Sounds with vibrato and no vibrato are presented within the folders with their annotations in cvs format. All annotations except than the alto saxophone recordings include derivative analysis of pitch parameters. The reason of choosing following subsets is that all subsets are created in MTG and they are open source. The MTG violin recordings includes single notes of two octaves and four different melodies starting from different pitch are recorded in wav format sampled at 44.1 kHz. Each sound separated by semitones is recorded four times for no vibra...
This proposed research investigates movement-based interaction techniques for designing new music... more This proposed research investigates movement-based interaction techniques for designing new musical instruments. It provides an aesthetics-based evaluation of a case study, Bodyharp, a new digital musical instrument at the intersection of music and dance. My dissertation examines not only developing new musical instruments but also performance practices that can be shared across diverse skill sets and abilities. In this project, I investigate this approach by bringing both the musician’s and the mover’s experiences closer to each other. In this paper, the design of the instrument and its earlier iterations are provided as well as an ongoing user study. The preliminary results of the user study, the future directions of this dissertation project, and its anticipated contributions are discussed.
Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, 2020
Videos are a convenient platform to begin, maintain, or improve a fitness program or physical act... more Videos are a convenient platform to begin, maintain, or improve a fitness program or physical activity. Traditional video systems allow users to manipulate videos through specific user interface actions such as button clicks or mouse drags, but have no model of what the user is doing and are unable to adapt in useful ways. We present adaptive video playback, which seamlessly synchronises video playback with the user's movements, building upon the principle of direct manipulation video navigation. We implement adaptive video playback in Reactive Video, a vision-based system which supports users learning or practising a physical skill. The use of pre-existing videos removes the need to create bespoke content or specially authored videos, and the system can provide real-time guidance and feedback to better support users when learning new movements. Adaptive video playback using a discrete Bayes and particle filter are evaluated on a data set collected of participants performing tai...
This research represents an evolution and evaluation of the embodied physical laptop instruments.... more This research represents an evolution and evaluation of the embodied physical laptop instruments. Specifically, these are instruments that are physical in that they use bodily interaction, take advantage of the physical affordances of the laptop. They are embodied in the sense that instruments are played in such ways where the sound is embedded to be close to the instrument. Three distinct laptop instruments, Taptop, Armtop, and Blowtop, are introduced in this paper. We discuss the integrity of the design process with composing for laptop instruments and performing with them. In this process, our aim is to blur the boundaries of the composer and designer/engineer roles. How the physicality is Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). NIME’19, June 3-6, 2019, Federal University of Rio Grande do Sul, Porto Alegre, Brazil. achieved by leveraging musical gestures gained through traditional instrument practi...
Mutualizing the body and the instrument offers a different way of thinking about designing embodi... more Mutualizing the body and the instrument offers a different way of thinking about designing embodied musical interfaces. This research presents the design of the BodyHarp, a wearable instrument that combines large bodybased gestures with the fine control of hand-based instruments. This reflects a desire to create a single interface that both captures expressive, dance-like, body movement as well as nuanced gestural interactions. The BodyHarp was not designed as a separate artifact, but rather it was crafted as an augmentation to the human body. This fusion seeks to enhance the sense of intimacy between the player and the instrument and carries a different type of aesthetic — like playing a traditional instrument (the harp) but as part of the body. In other words, the BodyHarp aims to capture creative body movement and placing it in an instrumental context. In doing so, we aim to blur the transition between two gestural mediums (dance and playing an instrument) by mutualizing them — o...
We present a musical interface specifically designed for inclusive performance that offers a shar... more We present a musical interface specifically designed for inclusive performance that offers a shared experience for both individuals who are hard of hearing as well as those who are not. This interface borrows gestures (with or without their overt meaning) from American Sign Language (ASL), rendered using low-frequency sounds that can be felt by everyone in the performance. The Hard of Hearing cannot experience the sound in the same way. Instead, they are able to physically experience the vibrations, nuances, contours, as well as their correspondences with the hand gestures. Those who are not hard of hearing can experience the sound, but also feel it just the same, with the knowledge that the same physical vibrations are shared by everyone. The employment of sign language adds another aesthetic dimension to the instrument –a nuanced borrowing of a functional communication medium for an artistic end.
Assessments of acoustics in spaces which music is performed typically consider a sense of richnes... more Assessments of acoustics in spaces which music is performed typically consider a sense of richness, involving reverberant and resonant qualities, and clarity, which describes the degree to which individual and sequential musical events retain their intelligibility as effected by these factors. Implicit, but not entirely understood, is how the delicate balance between richness and clarity that defines the quality of musical acoustics, effecting the perception of musical consonance and dissonance. Musical consonance and dissonance integrates a sensorial factor (that is, the sensation of tonal clarity or roughness) with a cognitive awareness, whether implicit or explicit, of how musical elements (such as intervals and chords) relate to one another both as simultaneities and sequentially in terms of stability or instability. Introduction Traditionally, a concurrent dissonant interval creates a tendency for resolution to a consonance which constitutes one of the fundamental bases for cen...
This article presents design and performance practices for movement-based digital musical instrum... more This article presents design and performance practices for movement-based digital musical instruments. We develop the notion of borrowed gestures, which is a gesture-first approach that composes a gestural vocabulary of nonmusical body movements combined with nuanced instrumental gestures. These practices explore new affordances for physical interaction by transferring the expressive qualities and communicative aspects of body movements; these body movements and their qualities are borrowed from nonmusical domains. By merging musical and nonmusical domains through movement interaction, borrowed gestures offer shared performance spaces and cross-disciplinary practices.Our approach centers on use of the body and the design with body movement when developing digital musical instruments. The performer's body becomes an intermediate medium, physically connecting and uniting the performer and the instrument. This approach creates new ways of conceptualizing and designing movement-base...
Inclusive musical instruments benefit from incorporating wearable interfaces into digital musical... more Inclusive musical instruments benefit from incorporating wearable interfaces into digital musical instrument design, creating opportunities for bodily felt experiences and movement-based interactions. In this article, we discuss the evolution of our inclusive design approach behind the design and performance practices of three wearable musical instruments. By focusing on the embodied, somatic, and tacit dimensions of movement-based musical interaction, we evaluate these case studies, combining the third and first-person perspectives. The design and implementation of the wearable sensing, utilizing the additive manufacturing techniques, are discussed for each instrument and its performer in specific cases of musical expression. This article further discusses how our approach integrates music performance as a crucial step into design and evaluation, utilizing these performance practices and such collaborative settings for improved diversity and inclusion. Finally, we examine how our design approach evolves from user-centered design to more participatory practices, offering people with diverse abilities a shared music performance space.
Inclusive musical instruments benefit from incorporating wearable interfaces into digital musical... more Inclusive musical instruments benefit from incorporating wearable interfaces into digital musical instrument design, creating opportunities for bodily felt experiences and movement-based interactions. In this article, we discuss the evolution of our inclusive design approach behind the design and performance practices of three wearable musical instruments. By focusing on the embodied, somatic, and tacit dimensions of movement-based musical interaction, we evaluate these case studies, combining the third and first-person perspectives. The design and implementation of the wearable sensing, utilizing the additive manufacturing techniques, are discussed for each instrument and its performer in specific cases of musical expression. This article further discusses how our approach integrates music performance as a crucial step into design and evaluation, utilizing these performance practices and such collaborative settings for improved diversity and inclusion. Finally, we examine how our d...
Cochlear implant (CI) users experience several challenges when listening to music. However, their... more Cochlear implant (CI) users experience several challenges when listening to music. However, their hearing abilities are greatly diverse and their musical experiences may significantly vary from each other. In this research, we investigate this diversity in CI users' musical experience, preferences, and practices. We integrate multisensory feedback into their listening experiences to support the perception of specific musical features and elements. Three installations are implemented, each exploring different sensory modalities assisting or supporting CI users' listening experience. We study these installations throughout semi-structured and exploratory workshops with participants. We report the results of our process-oriented assessment of CI users' experience with music. Because the CI community is a minority participant group in music, musical instrument design frameworks and practices vary from those of hearing cultures. We share guidelines for designing multisensory integration that derived from our studies with individual CI users and specifically aimed to enrich their experiences.
The fast-growing technology of additive manufacturing(AM), 3D printing, enables unique features i... more The fast-growing technology of additive manufacturing(AM), 3D printing, enables unique features in numerous design areas including musical instrument design. The recent advancements further introduce incorporating electronics into 3D printing. While still in the proof of concept phase, the ability to fabricate embedded sensors with 3D printing offers promising applications for musical instrument design. In this paper, embedding sensors and functional parts in the additive manufacturing process is discussed. We present the design process of embedded sensing for rapid prototyping digital musical instruments (DMI). The establishment of 3D models and operation sequences for different musical applications can be challenging since they vary depending on the electronic components. In order to address this problem, we provide a set of design and manufacturing specifications to customize the prototyping process for embedding structural electronics. The application of this hybrid method is de...
Vibrato Dataset is the extended version of the previous dataset. It consists of vibrato sounds fo... more Vibrato Dataset is the extended version of the previous dataset. It consists of vibrato sounds for five different instruments providing different pitch ranges of notes, melodies, articulations, scales and musical techniques as staccato and legato. It includes 295 note tracks with vibratos and corresponding non vibrato tones and 49 melody and articulation tracks. Data Collection It is organized according to the source of the recordings. Sounds with vibrato and no vibrato are presented within the folders with their annotations in cvs format. All annotations except than the alto saxophone recordings include derivative analysis of pitch parameters. The reason of choosing following subsets is that all subsets are created in MTG and they are open source. The MTG violin recordings includes single notes of two octaves and four different melodies starting from different pitch are recorded in wav format sampled at 44.1 kHz. Each sound separated by semitones is recorded four times for no vibra...
This proposed research investigates movement-based interaction techniques for designing new music... more This proposed research investigates movement-based interaction techniques for designing new musical instruments. It provides an aesthetics-based evaluation of a case study, Bodyharp, a new digital musical instrument at the intersection of music and dance. My dissertation examines not only developing new musical instruments but also performance practices that can be shared across diverse skill sets and abilities. In this project, I investigate this approach by bringing both the musician’s and the mover’s experiences closer to each other. In this paper, the design of the instrument and its earlier iterations are provided as well as an ongoing user study. The preliminary results of the user study, the future directions of this dissertation project, and its anticipated contributions are discussed.
Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, 2020
Videos are a convenient platform to begin, maintain, or improve a fitness program or physical act... more Videos are a convenient platform to begin, maintain, or improve a fitness program or physical activity. Traditional video systems allow users to manipulate videos through specific user interface actions such as button clicks or mouse drags, but have no model of what the user is doing and are unable to adapt in useful ways. We present adaptive video playback, which seamlessly synchronises video playback with the user's movements, building upon the principle of direct manipulation video navigation. We implement adaptive video playback in Reactive Video, a vision-based system which supports users learning or practising a physical skill. The use of pre-existing videos removes the need to create bespoke content or specially authored videos, and the system can provide real-time guidance and feedback to better support users when learning new movements. Adaptive video playback using a discrete Bayes and particle filter are evaluated on a data set collected of participants performing tai...
This research represents an evolution and evaluation of the embodied physical laptop instruments.... more This research represents an evolution and evaluation of the embodied physical laptop instruments. Specifically, these are instruments that are physical in that they use bodily interaction, take advantage of the physical affordances of the laptop. They are embodied in the sense that instruments are played in such ways where the sound is embedded to be close to the instrument. Three distinct laptop instruments, Taptop, Armtop, and Blowtop, are introduced in this paper. We discuss the integrity of the design process with composing for laptop instruments and performing with them. In this process, our aim is to blur the boundaries of the composer and designer/engineer roles. How the physicality is Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). NIME’19, June 3-6, 2019, Federal University of Rio Grande do Sul, Porto Alegre, Brazil. achieved by leveraging musical gestures gained through traditional instrument practi...
Mutualizing the body and the instrument offers a different way of thinking about designing embodi... more Mutualizing the body and the instrument offers a different way of thinking about designing embodied musical interfaces. This research presents the design of the BodyHarp, a wearable instrument that combines large bodybased gestures with the fine control of hand-based instruments. This reflects a desire to create a single interface that both captures expressive, dance-like, body movement as well as nuanced gestural interactions. The BodyHarp was not designed as a separate artifact, but rather it was crafted as an augmentation to the human body. This fusion seeks to enhance the sense of intimacy between the player and the instrument and carries a different type of aesthetic — like playing a traditional instrument (the harp) but as part of the body. In other words, the BodyHarp aims to capture creative body movement and placing it in an instrumental context. In doing so, we aim to blur the transition between two gestural mediums (dance and playing an instrument) by mutualizing them — o...
We present a musical interface specifically designed for inclusive performance that offers a shar... more We present a musical interface specifically designed for inclusive performance that offers a shared experience for both individuals who are hard of hearing as well as those who are not. This interface borrows gestures (with or without their overt meaning) from American Sign Language (ASL), rendered using low-frequency sounds that can be felt by everyone in the performance. The Hard of Hearing cannot experience the sound in the same way. Instead, they are able to physically experience the vibrations, nuances, contours, as well as their correspondences with the hand gestures. Those who are not hard of hearing can experience the sound, but also feel it just the same, with the knowledge that the same physical vibrations are shared by everyone. The employment of sign language adds another aesthetic dimension to the instrument –a nuanced borrowing of a functional communication medium for an artistic end.
Assessments of acoustics in spaces which music is performed typically consider a sense of richnes... more Assessments of acoustics in spaces which music is performed typically consider a sense of richness, involving reverberant and resonant qualities, and clarity, which describes the degree to which individual and sequential musical events retain their intelligibility as effected by these factors. Implicit, but not entirely understood, is how the delicate balance between richness and clarity that defines the quality of musical acoustics, effecting the perception of musical consonance and dissonance. Musical consonance and dissonance integrates a sensorial factor (that is, the sensation of tonal clarity or roughness) with a cognitive awareness, whether implicit or explicit, of how musical elements (such as intervals and chords) relate to one another both as simultaneities and sequentially in terms of stability or instability. Introduction Traditionally, a concurrent dissonant interval creates a tendency for resolution to a consonance which constitutes one of the fundamental bases for cen...
Uploads
Papers