JP2020502719A - Hands-free navigation of touch operation system - Google Patents

Abstract

Aspects of the invention allow for hands-free navigation of a touch-based operating system. The system can automatically interact with the touch-based operating system and generate a hands-free command associated with the touch-based command. Embodiments can utilize motion and / or voice input to facilitate hands-free interaction with a touch-based operating system. [Selection diagram] Fig. 1

Description

  Touch devices such as smartphones, tablets, and laptops are aspects of everyday life. For example, it has become common for people to organize and direct social interaction using applications on smartphones and tablets. In addition, companies often rely on touch devices, among other various applications, such as communicating with employees, monitoring work, and reviewing project data. Touch devices typically run a touch operating system (such as Android® or iOS®). It essentially relies on touch-based input to control interaction with the operating system. These devices are convenient and valuable, but all have the same limitations. In essence, the user needs to manually interact directly with the device.

  For example, touch-based operating systems currently rely primarily on virtual keyboards to accept text input. Virtual keyboards often have relatively small buttons that require you to enter words one character at a time, and entering a message even at a medium length can be time consuming and impractical . Some touch-based operating systems provide speech recognition for translating text into words, but such speech recognition often requires an Internet connection, which is not always available. In addition, even if speech recognition is available, it is generally limited to text input, and touch-based operating systems provide a mechanism for navigating the user interface within an application and navigating between multiple applications. Do not provide. For example, navigating a multi-page document in a touch-based operating system typically requires the user to touch the screen with a scroll bar to navigate horizontally or to "swipe" the screen to scroll. Request.

  However, a user may wish (or need) to use these devices during periods when manual touch interaction is not possible, difficult, or impossible. For example, many individuals may not have the ability to physically interact with a touch-sensitive device if they need or want to do so. Further, even if a user can physically interact with the touch-sensitive device, environmental restrictions may limit manual interaction with the device in a safe or comfortable manner. Further, it may be desirable to access the features of the touch-sensitive device while the user is engaged in a task that requires his or her hand.

  Previous attempts to solve these problems have often relied on highly specialized specialized devices or processes that provide a cumbersome and inflexible user experience. In addition, previous solutions required modification of source code from existing applications designed for touch-based operating systems so that the applications would be compatible with hands-free input sources. Also, it is not practical to change all touch-based applications to take advantage of touch-free solutions. Therefore, existing solutions could only work with some selected touch-based applications. Further, prior solutions did not allow interaction with the touch-based operating system itself, but instead relied on the operating system. As a result, conventional solutions generally require users to learn an entirely new operating environment, rather than enable interaction with existing and widely adopted touch-based operating systems. I was

  This summary is intended to introduce, in a simplified form, an excerpt of the concepts further described below in the Detailed Description section of the present disclosure. This summary is not intended to identify key or essential features of the claimed subject matter and is intended to be used as an aid in determining the scope of the claimed subject matter. It is not intended to be.

  To address these issues, the present invention is generally directed to systems and methods for providing hands-free navigation of a touch-based operating system. Further, aspects described herein facilitate hands-free interaction with touch-based operating systems and applications without requiring source code changes. That is, apart from implementing the systems and / or methods described herein, the user may switch to a different application or abandon the familiar touch-based operating system to enjoy the benefits of hands-free interaction. No need to do. Further, although not required, in embodiments, the user can customize the hands-free navigation to provide features tailored based on the needs and desires of the user.

  Accordingly, aspects of the technology described herein provide systems and methods that facilitate hands-free navigation of a touch-based operating system. In one aspect, a hands-free navigation system analyzes a touch-based user interface of a touch-based operating system to identify touch-based scrolling features, associate scrolling features with hands-free commands, and provide a touch-based user interface to a display to a user. Is presented. Then, as the system rotates, translates, or moves in 3D space, the system detects those movements and translates them into touch commands at the touch user interface of the touch operating system. In other words, the system converts the touch scroll function into a motion command, detects the motion, and converts the motion into a scroll command. In this way, a touch-based operating system can be converted to a hands-free operating system that can use motion-based user commands.

  In another aspect, a hands-free system analyzes a touch-based user interface of a touch-based operating system to identify a control dialog (such as an icon associated with a command), associates the control dialog with a keyword queue, and provides Display a touch-type user interface on the display. The system may then process the speech input, identify keyword cues in the speech input, and convert the keyword cues to touch commands in a touch user interface of an associated touch operating system. In other words, the system converts the touch-type command into a keyword queue, and when the user speaks, execution of the desired command is performed in the touch-type environment. In this manner, a touch-based operating system can be converted to a hands-free operating system that can utilize voiced user commands. To facilitate interaction with the user, in another aspect, the system may display a keyword cue overlay overlaid on the touch-based user interface. These overlays provide visual prompts to help publish keyword cues to facilitate selection of the desired control dialog by the user.

  In one aspect, the various hands-free input types disclosed herein may be used simultaneously or in combination with one another. For example, the system may respond to both motion-based and voice-based user commands simultaneously. Further, a method is provided for facilitating hands-free navigation using motion-based user commands, voice-based user commands, and motion-plus-voice-based user commands.

  The present disclosure is described in detail below with reference to the accompanying drawings.

FIG. 1 is a diagram illustrating an exemplary computing environment suitable for use in implementing embodiments of the present disclosure. FIG. 2 is a diagram illustrating an exemplary hands-free navigation system according to aspects of the present disclosure. FIG. 3 is a diagram illustrating the exemplary hands-free navigation system shown in FIG. 2 according to a further aspect of the present disclosure. FIG. 4A illustrates an exemplary motion-based hands-free interaction method according to aspects of the present disclosure. FIG. 4B illustrates an exemplary spoken hands-free interaction method according to aspects of the present disclosure. FIG. 4C illustrates an exemplary motion-based and audio-based hands-free interaction method according to aspects of the present disclosure. FIG. 5 is a diagram illustrating an exemplary method for determining a multi-axis motion interaction in accordance with aspects of the present disclosure. FIG. 6A is a diagram illustrating an example user interaction according to aspects of the present disclosure. FIG. 6B is a diagram illustrating an example user interaction according to aspects of the present disclosure. FIG. 7A is a diagram illustrating an exemplary method for determining audio input according to aspects of the present disclosure. FIG. 7B illustrates an enlarged portion of the method of FIG. 7A, illustrating an exemplary method for determining spoken input, according to aspects of the present disclosure. FIG. 8A illustrates an example method of identifying a control dialog in an example touch-based user interface, according to aspects of the present disclosure. FIG. 8B is a diagram illustrating an example keyword cue overlay associated with a touch control dialog in an example touch user interface according to aspects of the present disclosure. FIG. 9 is a diagram illustrating an example head-mounted computing device having an example reference frame, according to aspects of the present disclosure. FIG. 10 is a diagram illustrating an exemplary computing device according to aspects of the present disclosure.

  The subject matter of this disclosure has been described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the patent. Rather, we claim that the claimed subject matter, including different steps or combinations of steps similar to those described herein, as well as other current or future technologies, It can be embodied in the following way. Further, although the terms "step" and / or "block" may be used herein to indicate different elements of the method used, the order of the individual steps is explicitly described. Except where otherwise, it should not be construed as indicative of any particular order between or within the various steps described herein. Each of the methods described herein may include a computing process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. The method may also be embodied as computer-usable instructions stored on a computer storage medium. The method may be provided by a stand-alone application, service or hosted service (stand-alone or in combination with other hosted services), or a plug-in to another product, to name a few.

  In general, aspects herein relate to mechanisms that enable hands-free interaction with a touch-based operating system. As will be appreciated, touch-based operating systems (such as Android®, iOS®) rely on input received via a touch screen of a mobile device running a touch-based operating system. There are many. However, interaction with the touch screen may be impossible or undesirable depending on the capabilities of the user and the circumstances in which the mobile device is used. For example, when a user is using his or her hand to perform another task, interrupting that task and interacting with the touch-based operating system is often inconvenient and time consuming. Accordingly, the embodiments relate to devices, methods, and systems that facilitate hands-free interaction with a touch-based operating system.

  One aspect herein relates to a head-mounted computing device (such as a headset) that implements a method for hands-free interaction with a touch-based operating system. In a non-limiting example, the headset includes a display (head-up display, eyepiece display, etc.), sensors (camera, microphone, accelerometer, gyroscope, magnetometer, etc.), one or more processors, and memory. Can be included. The headset may be communicatively coupled to a mobile device running a touch-based operating system. Also, the headset may be configured to display an instance of a touch-based operating system user interface, for example, on an eyepiece display of the headset. Further, the headset can provide a hands-free interaction mode that facilitates interaction with a touch-based operating system user interface using hands-free input received via the sensor. For example, one exemplary form uses movement detected by the headset to determine a command or command to navigate to perform an operation of the touch-based operating system.

  Another example aspect uses voice commands detected by the headset to determine instructions or commands to perform an operation of the touch-based operating system. Accordingly, the headset also monitors included sensors (such as microphones, among others), analyzes the inputs received by the sensors, determines touch operating system commands or commands associated with the inputs, May be configured to execute commands or commands for navigating within the user interface. Further, the headset is configured to use a headset component or software module to analyze the touch operating system user interface and code associated with the touch operating system in real time to facilitate a hands-free interactive mode. Can be done.

  Another exemplary aspect is a voice command and movement detected by a headset to navigate a user interface of a touch-based operating system and determine commands or commands to perform operations of the touch-based operating system. Use both. In addition, the headset uses environment-specific data (user-specific settings, hands-free interface-specific settings, keyword queue library, touch-type interface-specific settings, location-specific settings, to improve the usability of the hands-free interactive mode. Etc.) or may communicate with the database. The headset may include a wireless communication system (Bluetooth (registered trademark), NFC, RFID, WIFI (registered trademark), etc.) to enhance the usability of the hands-free interactive mode. By way of example, a wireless communication system may enhance the usability of a hands-free interactive mode by providing a headset with location information that may be correlated with environment-specific data stored in a customized database. The headset may be communicatively coupled to a mobile device having a wireless communication system to enhance the usability of the hands-free interactive mode. Further, the headset may be configured to utilize a networked, customized database containing environment-specific data.

  Referring now to FIG. 1, a block diagram is provided illustrating an exemplary operating environment 100 that may be employed in some embodiments of the present disclosure. It should be understood that this and other configurations described herein are described by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, and groupings of functions, etc.) may be used in addition to or in place of those shown, some elements having clarity, May be omitted entirely. In addition, many of the elements described herein are functional entities that can be implemented as discrete components or distributed components, or with other components, and in any suitable combination and location. Various functions described herein as being performed by one or more entities can be performed by hardware, firmware, and / or software. For example, some functions may be performed by a processor that executes instructions stored in memory.

  Although other components are not shown, the exemplary operating environment 100 includes several user devices, such as user devices 102a-102n, several data sources, such as data sources 104a and 104b through 104n, a server 106, a sensor 103a. To 103 n and the network 110. It should be understood that the environment 100 shown in FIG. 1 is an example of a suitable operating environment. Each of the components shown in FIG. 1 may be implemented via any type of computing device, such as, for example, computing device 1000 described in connection with FIG. These components can communicate with one another via a network 110, which can include, but is not limited to, one or more local area networks (LANs) and / or wide area networks (WANs). In the exemplary embodiment, network 110 includes the Internet and / or a cellular network, including various possible public and / or private networks.

  It should be understood that any number of user devices, servers, and data sources may be employed within operating environment 100 within the scope of the present disclosure. Each may include a single device or multiple devices working together in a distributed environment. For example, server 106 may be provided via a plurality of devices located in a distributed environment that collectively provide the functionality described herein. In addition, other components not shown may be included in the distributed environment.

  User devices 102a-102n may comprise any type of computing device that can be used by a user. For example, in one embodiment, user devices 102a-102n may be computing devices of the type described with respect to FIG. 10 herein. By way of example, and not limitation, user devices include personal computers (PCs), laptop computers, mobile or mobile devices, smartphones, tablet computers, smart watches, wearable computers, personal digital assistants (PDAs), MP3 players, global positioning systems (GPS) or device, video player, handheld communication device, gaming device or system, entertainment system, in-vehicle computer system, embedded system controller, camera, remote control, barcode scanner, computerized measuring device, equipment, consumer electronics , Workstations, head-mounted computing devices, or these depicted devices The combination of meaning, or any other suitable device.

  User devices 102a-102n can be client devices on the client side of operating environment 100, while server 106 can be the server side of operating environment 100. Server 106 may comprise server-side software designed to work with client-side software of user devices 102a-102n to implement any combination of the features and functions described in this disclosure. This division of operating environment 100 is provided to illustrate one example of a suitable environment, and it is not a requirement for each embodiment that any combination of server 106 and user devices 102a-102n be a separate element. .

  Data sources 104a and 104b through 104n may include data sources configured to make data available to any of the various components of operating environment 100, or to hands-free interaction system 200 described in connection with FIG. And / or a data system. For example, in one embodiment, one or more data sources 104a-104n are provided (or made accessible) to storage 270 of FIG. The data sources 104a and 104b through 104n may be separate from the user devices 102a-102n and the server 106 or may be incorporated therein and / or integrated therewith. In one embodiment, one or more of data sources 104a-104n may be integrated with or associated with one or more of user devices 102a-102n or server 106. It has a sensor. The operating environment 100 includes components of the hands-free interaction system 200 described in FIGS. It can be used to implement one or more of them.

  Referring now to FIG. 2, a block diagram is provided illustrating an exemplary embodiment of a hands-free interaction system 200 that may be employed in some embodiments of the present disclosure. The hands-free interaction system 200 generally operates to facilitate hands-free interaction with applications and features of the touch-based operating system 202. It should be understood that the hands-free interaction system shown in FIG. 2 is an example of one system that can be employed in embodiments of the present disclosure. Each illustrated component may include one or more computing devices similar to the operating environment 100 described with reference to FIG. The hands-free interaction system 200 should not be interpreted as having any dependency or requirement relating to any single module / component or combination of modules / components illustrated. For example, the hands-free interaction system 200 may include multiple devices configured in a distributed environment that collectively provide the functionality described herein. It should be understood that the hands-free interaction system 200 and / or its various components may be located anywhere according to various embodiments of the present disclosure.

  Head-mounted computing device 220 (described in more detail with reference to FIG. 9) generally facilitates hands-free interaction with touch-based user interface 206 of touch-based operating system 202. Head-mounted computing device 220 may include the input / output components of various headset devices, such as motion sensors, audio sensors, displays, and input controls, among others. Further, head-mounted computing device 220 may include computer-enabled instructions stored on a computer storage medium, such as storage 270. Accordingly, the head-mounted computing device 220 can be configured to perform a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor (eg, headset processor 280) that executes instructions stored in memory. Such methods may be provided by stand-alone applications, services or hosted services (stand-alone or in combination with other hosted services), or plugins to other products, to name a few.

  The functions and processes performed by hands-free interaction system 200 may be associated with an application, service, or routine (such as headset application 276). In particular, such applications, services, or routines may run on head-mounted computing device 220 or may be distributed across multiple devices. For example, the functions and processes described herein may be executed on a touch-based user device (such as the user device 102a), a server (such as the server 106), or may be implemented in the cloud. . Further, in some embodiments, components of hands-free interaction system 200 may be distributed across network 110. In addition, these components, the functions performed by these components, or the services performed by these components, are implemented in the appropriate abstraction layers, such as the operating system, application, and hardware layers of the computing system. Can be done. Of computing systems). Alternatively or additionally, the functions of these components described herein and / or embodiments of the invention may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), and on chip systems. Systems (SOCs), complex programmable logic devices (CPLDs). Further, although the functionality has been described herein with respect to certain components illustrated in the exemplary hands-free interaction system 200, in some embodiments, these components are shared with other components. Or distributed.

  Accordingly, head-mounted computing device 220 may include one or more headset processors 280 that execute instructions (which may be stored by headset application 276) to provide a hands-free interaction mode. The hands-free interaction mode can facilitate interaction with programs, applications, and features of the touch-based operating system 202 via the head-mounted computing device 220. In one form, the headset application 276 includes instructions to facilitate hands-free interaction with existing binary applications running on the touch-based operating system 202. For example, the hands-free interaction system 200 may be configured to be used with any number of applications through publishers or existing binaries, such as applications available from Playstore, Appsstore, and other sources of touch-based applications. Can be done. In addition, the headset engine 210 queries the application 208 running on the touch operating system 202 to determine screen components / functions, such as input controls, form elements, and navigation commands included in the touch user interface, among others. I can do it. For example, headset engine 210 may analyze the user interface layer of touch-based operating system 202 to determine when various screen components / functions are being provided for display. In this manner, when the touch-based application is running, various displayed UI components of the application can be determined. As described below, screen components / features can be extracted or identified and provided to other components of the hands-free interaction system 200 for processing. As a result, virtually any application operable on the touch-based operating system 202 can be enabled at runtime for hands-free interaction. Further, the hands-free interaction system 200 may include XML files for applications that are not compatible with standard hands-free interaction mode commands. The XML file may override the standard hands-free interactive mode instructions and provide customized instructions. In addition, the XML file may be mixed with the existing binaries of the application 208 at runtime so that the existing binaries need not be modified.

  Further, head-mounted computing device 220 may include various headset device I / Os 222, for example, components that can sense or detect hands-free input received via head-mounted computing device 220. The received input is processed, for example, by hands-free input determiner 240 to generate one or more hands-free commands. Further, hands-free interaction system 200 may be configured to determine and / or generate a command associated with the received hands-free input. The generated hands-free command may be communicated (eg, by communication component 232) to touch-based operating system 202 for execution. The determined command can programmatically instruct the touch-based operating system 202 to execute the command as if it were executing the corresponding touch-based input.

  The headset engine 210 generally includes a touch operating system 202, a touch user interface 206, a motion processing system 242, a sound processing system 250, an interface analyzer 212, storage 270, a headset device input / output (I / O) 222. , And other respective sub-components. In some forms, headset engine 210 initiates a hands-free interactive mode in response to receiving a signal from headset device I / O 222. For example, a physical input control 238 (such as a button, switch, etc.) can receive an input to initiate a hands-free interaction mode. In some aspects, the headset engine 210 initiates an analysis of the touch-based user interface 206 and / or the touch-based operating system 202, such as the determined touch-sensitive scrolling features and touch-sensitive control dialogs. In some forms, headset engine 210 receives motion data (eg, from sensors 226a-226n) and sends information to hands-free input determiner 240. In some embodiments, headset engine 210 receives audio input 224 from headset device I / O 222 and sends information to hands-free input determiner 240.

  Touch user interface 206 generally facilitates user interaction with touch operating system 202 in hands-free interaction system 200. In some forms, the touch-based user interface 206 may include a touch-based scrolling function (“swipe” function, horizontal scroll bar, vertical scroll bar, etc.). In some forms, the touch-sensitive user interface 206 includes a touch-sensitive control dialog (eg, a text box or field, a check box, an application icon, a document tool, a radio button, etc.).

  In some forms, storage 270 may include keyword custom library 272. The keyword custom library 272 may include a database containing keyword cues for touch control dialog associations. In one form, storage 270 may also include user-specific settings, preferences, thresholds, permissions, or any data associated with an individual or group of individuals. In one form, storage 270 may be headset application 276. Storage 270 may be communicatively coupled to any components and sub-components of hands-free interaction system 200.

  Audio input 224 generally refers to a component for capturing audio, such as a microphone (eg, a directional and omni-directional microphone). In embodiments, the audio input 224 is located at various points on the head-mounted computing device 220 that is configured to capture ambient noise and ultimately assist in processing and analyzing user audio input. May be included. It will be appreciated that the audio input 224 can be any sensor or system of sensors that can sense the audio input and convert the audio input into an audio feed without departing from the scope of the present disclosure. Audio output 230 generally facilitates sound output to the user. It will be appreciated that any audio output component (such as a speaker) capable of producing a sound in response to an electrical input may be used in embodiments without departing from the scope of the present disclosure. In embodiments, the audio output 230 may be configured to communicate with at least the headset device I / O 222. Communication component 232 generally facilitates communication between head-mounted computing device 220 and other devices via any suitable communication protocol. In embodiments, the communication component may comprise a wireless communication system as described above or below with reference to FIG.

  Display 234 generally facilitates the visual presentation of data to a user. It will be appreciated that any display may be used in various embodiments without departing from the scope of the present disclosure. Sensors 226a-226n may include cameras, microphones, GPS, RFID sensors, infrared sensors, optical sensors, magnetometers, gyroscopes, capacitive transducers, potentiometers, resistance transducers, synchro, accelerometers, and micro gyroscopes, among others.

  Referring now to FIG. 3, a block diagram is provided that illustrates additional aspects of the hands-free interaction system 200, where an exemplary head-mounted computing device 220 may employ some embodiments of the present disclosure. it can. Head-mounted computing device 220 may employ embodiments of the present disclosure such as motion capture, motion analysis, audio input, audio analysis, audio output, image capture, position detection, orientation determination, environment determination, interface display, A system for location and networking, and user devices 102a-102n that can be communicatively coupled to the head-mounted computing device 220 and communicatively coupled to the network 110, with respect to FIG. And a data source 104a. The components of the hands-free interaction system 200 may be one or more, such as a set of compiled computer instructions or functions, program modules, computer software services, or a computing device described in connection with FIG. It can be implemented as a process configuration executed on a plurality of computer systems.

  In one embodiment, the functions performed by the components of the hands-free interaction system 200 relate to translating the displacement into a touch command, a command, or an input in a touch operating system. In embodiments, the hands-free interaction system 200 may detect a headset input / output (I / O) 222 of the head-mounted computing device, a headset engine 210, and a system or subsystem within the hands-free interaction system. , Processing, distributing, monitoring, and / or operating. For example, in response to receiving the activation command, the headset engine 210 initiates a hands-free interactive mode on the head-mounted computing device, wherein the hands-free interactive mode interacts with a touch-based operating system (OS) user interface. May be enabled. As part of initiating the hands-free interaction mode, the headset engine 210 can activate the interface analyzer 212, the motion processing system 242, the sound processing system 250, and the environment analyzer 214.

  Motion processing system 242 generally facilitates processing of motion-based input data detected by headset device I / O 222. Motion processing system 242 may reside at head-mounted computing device 220, user devices 102a-102n, server 106, or any location that may be communicatively coupled to hands-free interaction system 200 via, for example, network 110. . In embodiments, the motion processing system 242 may be a subsystem of a headset engine. In embodiments, the motion processing system 242 may be a subsystem of one or more user devices 102a-102n communicatively coupled to the head mounted computing device 220 via the communication component 232. In other embodiments, motion processing system 242 may be a subsystem of one or more network devices communicatively coupled to head-mounted computing device 220 via communication component 232.

  Thus, data from sensors 226a-226n may be sent to motion processing system 242 for analysis. In some embodiments, the motion processing system 242 may include sub-components that include a motion detection component 244, a calibration control component 246, and a motion conversion component 248 (described in further detail below). In an aspect, the motion processing system 242 may be communicatively connected to the hands-free interaction system 200 via the communication component 232. Lateral, translational, and rotational movements of the headset are detected by sensors 226a-226n and processed by motion processing system 242 to determine neutral position and orientation. For example, the sensor data may be processed by the motion processing system 242 to detect the position and orientation of the headset with nine degrees of freedom about three axes. In embodiments, the motion processing system 242 may calibrate the neutral position during the start of the hands-free interaction mode by detecting the orientation of the headset at the start of the hands-free interaction mode.

  In an embodiment, the threshold displacement may be a predetermined displacement value from a neutral position, such as rotating the headset +/− 10 degrees from neutral in any axis. In addition, the threshold displacement may be an angular change, translation, rotation, or any other movement of the head mounted computing device 220. As can be appreciated, the description generally refers to the singular, but any number of threshold displacements can be determined. The threshold displacement may correspond to any number of touch-based inputs for interacting with the touch-based operating system 202. For example, a scroll-up touch-type input may have a corresponding hands-free input as turning the head-mounted computing device 220 upward. Thus, the hands-free rotation input may have a corresponding threshold of a predetermined degree above the neutral position. As a result, careless hands-free input can be reduced or eliminated.

  In other forms, the threshold displacement may be adjusted automatically and iteratively, for example, by the motion processing system 242 and / or the headset engine 210. By way of example, and not limitation, hands-free interaction system 200 may include a computer-learning or machine monitoring hands-free input (which may be stored in user data 274) to adjust the threshold displacement based on previous headset motion data. A learning instruction can be included. In other aspects, the threshold displacement may be automatically adjusted based on usage data associated with any number of locations associated with the head mounted computing device 220 (eg, may be determined by the position determiner 216). In another aspect, the displacement threshold may be a customized value, which may be determined by a user. For example, the user can adjust the threshold displacement setting via the user interface of the hands-free interaction application, which is stored in the hands-free interaction system 200 and executed by any of the devices described herein. obtain. Accordingly, motion processing system 242 may compare the detected displacement, or translation of the headset, to a threshold displacement, as described in more detail below.

  Motion detection component 244 may monitor motion processing system 242 to detect displacement of head-mounted computing device 220. For example, the motion detection component 244 includes an initial reference image stored by the calibration control component 246 and an initial of the head mounted computing device 220 captured by the motion processing system 242 to detect displacement of the head mounted computing device 220. A subsequent image for the location can be compared. It will be appreciated that any number of image analysis algorithms can be used to detect displacement of the head mounted display with respect to the initial position of the head mounted computing device 220 by comparing the initial reference image with a subsequent image. Further, the threshold and / or detected displacement may include determining a duration of the displacement. For example, a threshold displacement may require that the displacement be sustained for 5 seconds to be recognized as a hands-free input. Further, any type of data from a motion detection sensor (such as sensors 226a-226n, which may include accelerometers, gyroscopes, etc., as described herein) can be used to measure or detect displacement. Please understand what you can do.

  When the motion detection component 244 detects a displacement of the head-mounted computing device 220 that exceeds a threshold displacement, the motion conversion component 248 can convert the angular displacement into an instruction corresponding to one or more touch-based inputs. Motion conversion component 248 can determine an appropriate command based on the type of displacement. In embodiments, the motion transform component 248 may use a modifier to augment the command. For example, the page scroll command may be reinforced by a plurality of page correction units, such as a 10-page scroll. The modifier may be based on a feature associated with the detected hands-free input, such as a duration of the detected displacement.

  Sound processing system 250 generally facilitates processing of audio input data detected by headset device I / O 222. Accordingly, data from sensors 226a-226n may be sent to sound processing system 250 for analysis. In some forms, data from the audio input 224 may be sent to the sound processing system 250 for analysis. In some embodiments, the sound processing system 215, 250 may include sub-components including a sound detection component including an interface analyzer 212, 252, a sound processing component 254, and a sound conversion component 256. In some forms, the sound processing system 250 can compare the keyword cues associated with the touch control dialog with the detected speech input. In some forms, sound processing system 250 may be communicatively connected to hands-free interaction system 200 via communication component 232. Accordingly, sound processing system 250 may be located at a user device, a network, a server, or any location that is communicatively connected to hands-free interaction system 200.

  In embodiments, the touch-based operating system may further include a sound sharing component (not shown) because it is configured to allow voice input to only one application of the sound processing system. The audio sharing component may enable multiple processes, applications, components, etc. to receive audio input simultaneously. In other words, the audio sharing component may allow the audio feed to continue to the touch-based operating system and to the audio detection component 252 without further processing or analysis. Stated another way, the audio sharing component facilitates providing audio feeds to the touch-based application without compromising the functionality of the sound processing system.

  For example, a user can simultaneously run a touch-based operating system and a teleconferencing application in a hands-free navigation system, and an audio sharing component can allow audio feeds to continue to the teleconferencing application. . Further, the audio sharing component can provide an audio feed to the audio detection component 252. Thus, in embodiments, the audio sharing component can duplicate the audio feed.

  The audio detection component 252 generally facilitates monitoring the audio input 224 and / or the sensors 226a-226n to detect an audio feed. For example, the audio detection component 252 may listen to the microphone of the head mounted computing device 220 to detect that a signal is being received by the microphone. Continuing with this example, the audio detection component 252 may be responsible for determining that the signal received by the microphone is above a predetermined volume, which is used to determine if the signal is a hands-free audible input. It may indicate that further processing is required. In embodiments, the audio detection component 252 provides the detected audio feed to the audio processing component 254.

  The audio processing component 254 generally facilitates processing of the audio feed to identify, separate, and analyze user utterances. In embodiments, the speech processing component 254 may use speech recognition algorithms, noise reduction algorithms, speech-sentence algorithms, machine learning algorithms, etc. to process the speech feed. In some forms, the audio processing component 254 can receive multiple audio feeds from the audio detection component 252. In these embodiments, the audio processing component 254 may process multiple audio feeds to at least partially separate the user's audio from background noise. It will be appreciated that any noise reduction algorithm, speech separation algorithm, or any suitable algorithm or technique may be used to at least partially separate the user's voice from the background. In embodiments, the audio processing component 254 may receive the audio feed from the audio detection component 252 and identify the first audible input associated with the touch-based control dialog by the interface analyzer 212. In an embodiment, the audio processing component 254 analyzes the audio feed and compares the audio feed with the keyword queue to determine whether the processed audio feed matches the keyword queue.

  The audio conversion component 256 generally facilitates the conversion of the audio feed-keyword cue match into an associated control dialog. Accordingly, the speech conversion component 256 can receive a matching keyword cue from the speech processing component 254 and determine the control dialog associated with the keyword cue.

  The interface analyzer 212 generally detects the detection of the touch user interface 206, the touch operating system 202, the application 208 in the touch user interface 206, and the touch user interaction in the document 204 in the touch user interface 206. Facilitate. As used herein, touch-based user interaction features include touch-based scrolling features (“swipe” feature, horizontal scroll bar, vertical scroll bar, etc.), touch-based control dialogs (text boxes or fields, checkboxes, application icons). , Writing tools, radial buttons, etc.) and any more common elements, functions, icons, commands, codes, etc. that facilitate user interaction within the touch-based user interface, operating system, applications, and / or documents. Including extensions, macros, etc. In an aspect, the interface analyzer 212 detects and detects touch user interactivity by scanning the source code of the touch user interface 206, the touch operating system 202, and / or the application 208 within the touch user interface 206. And / or may be identified. In an aspect, interface analyzer 212 may refer to keyword custom library 272 and / or user data 274 to facilitate environment-specific functions.

  In some embodiments, environment analyzer 214 facilitates the analysis of environmental data and environment-specific characteristics of generally hands-free interaction system 200. Environmental data may be any data related to the operation of headset engine 210 or its subcomponents. By way of non-limiting example, environment data can be user data (such as user data 274), application data (such as associated with application 208), or data received from communication component 232 or location determiner 216. . In an embodiment, environment analyzer 214 further monitors interface analyzer 212 to determine whether the customized data is associated with the current instance of the touch-based user interface. In an embodiment, the environment analysis unit may change the function of the hands-free input determination unit 240, the headset engine 210, or their respective subcomponents in response to the analysis. For example, in response to the interface analyzer 212 analyzing a particular instance of the touch-based user interface, the environment analyzer 214 may determine a custom keyword queue library (such as the custom keyword library 272) associated with the particular instance of the touch-based user interface. ) Can be placed. The environment analyzer 214 may then communicate the custom keyword cue library to the sound processing system 250

  In an embodiment, the environment analysis unit 214 may use the location information to change the function of the hands-free input determination unit 240. For example, the environment analysis unit 214 can analyze position data (such as Bluetooth beacon information detected by the position determination unit 216) associated with a specific custom keyword library. In embodiments, the environment analyzer 214 may determine that a particular motion-based threshold is indicated as a result of the environment data.

  In embodiments, the interface analyzer 212 detects a compatible touch-based user interface, for example, a Google Android or Apple iOS, analyzes the touch-based OS, and detects touches associated with the first instance of the user interface. Detect expression commands. The interface analyzer may, for example, detect that the first instance of the user interface comprises a touch scroll function. For example, the touch scroll function may be associated with an application for guiding to a different menu screen of the touch operating system, a touch scroll function associated with the touch operating system, for guiding to a different menu screen of the application. A touch-sensitive scrolling function and / or a touch-sensitive scrolling function associated with the document for navigating to different parts of the document may be provided.

  In an embodiment, when the interface analyzer 212 detects a touch scroll function, the headset engine 210 calibrates the motion capture system, monitors the displacement, and converts the displacement to an associated touch scroll function 242. Start When activated, the motion processing system 242 may activate the calibration control component 246. The calibration control component 246 detects the initial position of the head mounted computing device 220. This initial position includes the orientation of head-mounted computing device 220 with respect to one or more axes. In embodiments, the calibration control component 246 determines an initial position of the head mounted computing device by activating the motion detection component 244 using a camera (eg, one of the sensors 226a-226n). Can be. The calibration control component 246 may at least temporarily store the image as an initial reference image for comparison with a subsequent image to measure the relative movement of the head mounted computing device 220.

  Further, in embodiments, the interface analyzer 212 associates the detected touch-based scroll function with a displacement of the head-mounted computing device 220 in a corresponding direction. For example, when the interface analyzer 212 detects a touch scroll function associated with a touch OS corresponding to a “swipe” from left to right in the first instance of the user interface. The interface analyzer 212 may associate the rightward displacement of the head mounted computing device 220 with respect to the initial position with a touch swipe from left to right. In another example, if the interface analyzer 212 detects a vertical scroll bar in the first instance of the user interface, the interface analyzer 212 determines the angular displacement “up” with respect to the initial position of the head-mounted computing device 220, This can be associated with moving the vertical scroll bar up a predetermined number of scroll units. When the interface analyzer 212 detects a horizontal scroll bar in the first instance of the user interface. The interface analyzer 212 can associate the rightward angular displacement of the head mounted computing device 220 with respect to the initial position to move the horizontal scroll bar rightward by a predetermined number of scroll units.

  Further, the headset engine 210 can invoke the display component 234 to display a first instance of the touch-based operating system user interface on the display of the head-mounted computing device 220 (see FIG. 2). Described in more detail and indicated by reference numeral 234). In embodiments, the headset engine 210 may then invoke a motion detection component 244 that detects a first angular displacement of the head mounted computing device. The first angular displacement is greater than a first threshold angular displacement, and the first threshold angular displacement is an angular displacement with respect to an initial position.

  Further, the interface analyzer 212 may include a touch-sensitive scroll function corresponding to the left-to-right swipe and the right-hand angular displacement relative to the initial position of the head-mounted computing device associated with the left-to-right touch-type swipe. The motion detection component 244 detects the angular displacement of the head-mounted computing device from the initial position to the right above a threshold, and the motion conversion component 248 determines that a command to swipe from left to right is required. can do. Next, the motion conversion component 248 can convert the angular displacement into a command corresponding to a swipe from left to right, and pass the command to the command generator 213. The command generator 213 generates a touch input corresponding to the command indicated by the motion conversion component 248, and executes the command in the touch user interface. It will be appreciated that any displacement may be detected, analyzed, and converted by the system, and the foregoing examples are intended as examples and not as limitations.

  In an embodiment, the interface analyzer 212 detects a compatible touch operating system, analyzes the touch user interface, and detects at least one touch control dialog associated with the first instance of the user interface. I do. For example, the touch control dialog may include a touch command associated with a touch operating system to enable audio output or change the audio output volume, and / or, for example, to launch an application. Alternatively, there may be a touch control dialog associated with the application to select an application element, text field or "send" element. It will be appreciated that the above examples are only a few of the potential touch control dialogs and are not intended to be limiting.

  In embodiments, when the interface analyzer 212 detects a touch control dialog, the headset engine 210 monitors and processes the audio input, analyzes the audio input, and converts the audio input into an associated touch control dialog. The processing system 250 can be activated. When activated, the sound processing system 250 activates the interface analyzer 212 to detect and identify control dialog options and associate keyword cues. In embodiments, the interface analyzer 212 detects the touch-based control dialog by scanning source code associated with the first instance of the user interface at runtime to extract features. For example, if the interface analyzer 212 detects a "button" embedded in the first instance of the user interface, the interface analyzer 212 associates the button text with the keyword queue. In an embodiment, the interface analyzer 212 may detect a customized keyword queue library associated with the first instance of the user interface stored in a hands-free computing device memory (such as the storage 270). . In embodiments, the interface analyzer 212 may detect a customized keyword queue library stored on a communicatively coupled user device (such as the user device 102a). In embodiments, the interface analyzer 212 may detect a customized keyword queue library stored in a communicatively coupled data source (such as the data source 104a).

  The headset engine 210 activates the display component 234 to display a first instance of a touch-based operating system user interface on a display of the head-mounted computing device. In embodiments, the headset engine 210 then invokes the audio detection component 252 to detect audio received, for example, via the sensors 226a-226n or the audio input 224, and to convert the audio input to the audio processing component 254. Pass to As used herein, an audio feed may refer to either an acoustic signal captured by an audio input device or an electrical signal generated by an audio input element.

  In response to command generator 213 executing the command in the touch-based user interface, headset engine 210 then directs display component 234 to print a second instance of the touch-based operating system user interface. Display on the display of the mount computing device.

  Referring now to FIG. 4, a block diagram is provided illustrating an exemplary motion-based hands-free interaction mode 400 that may be implemented, at least in part, by the headset described with reference to FIG. In an embodiment, the headset engine initiates a hands-free interaction mode 402 associated with the motion processing system 242. The hands-free interaction mode 402 may include launching an interface analyzer to detect a touch scroll function in a first instance of a user interface and associate the touch scroll function with a first angular displacement. At block 404, an initial position of the hands-free computing device is detected and an initial reference orientation is determined. In embodiments, this may be performed by the calibration control component 246, as described with reference to FIG. In embodiments, the calibration control component 246 may be restarted by the user at any time to reset the reference orientation. For example, the user may be headed to a strange or unpleasant position where it would not be desirable to maintain that position during hands-free navigation once the initial reference orientation has been determined, , The calibration control component 246 may be restarted so that the reference orientation may be re-determined. In embodiments, the head-mounted computing device 220 may have a button (such as a physical input element 238) associated with restarting the calibration control component 246. Additionally or alternatively, in one embodiment, a predetermined voice command is associated with restarting the calibration control component.

  At block 406, the display shows a first instance of the user interface. In an embodiment, this is performed by the headset engine 210 as described with reference to FIG. At block 408, movement of the head mounted computing device is detected by the motion detection component 244, as described with reference to FIG. At block 410, the detected movement is determined by the motion detection component 244 to have exceeded an angular threshold associated with the touch scrolling function, as described with reference to FIG. At block 412, the detected displacement is converted by the motion conversion component 248 into a touch scroll command associated with the touch scroll function. Further, a command is generated and executed within the first instance of the touch-based interface. At block 414, a second instance of the user interface is displayed in response to execution of the touch scroll command. It will be appreciated that the method 400 can be performed in an iterative manner as many times as necessary.

  In an embodiment, some processes of the motion-based hands-free interaction mode may include a user device (such as user device 102a) communicatively connected to the head-mounted computing devices 302, 220 as described with reference to FIG. May be at least partially completed.

  Referring now to FIG. 4b, there is provided a block diagram illustrating an exemplary spoken hands-free interaction mode 416 that may be implemented at least partially by the headset shown in FIG. In an embodiment, the headset engine initiates a hands-free interactive mode 418 associated with the sound processing system 250. Initiating the hands-free interaction mode 418 may include activating the interface analyzer to detect at least one touch-based control dialog within the first instance of the user interface. At block 420, at least one touch control dialog is identified and associated with a keyword cue. In embodiments, the interface analyzer may analyze the source code of an application running in the touch-based user interface and identify a name associated with the touch-based control dialog in the application source code. The interface analyzer may then generate a keyword cue substantially similar to the name from the application source code and associate the generated keyword cue with the touch control dialog. For example, if the touch-based user interface displays a structural view of the application and the application has a touch-based control dialog that opens a zoom function, the interface analyzer can access the application source code in real time and use the zoom function. Identify the part of the code that is encoding, detect that the function is named "Zoom", generate the keyword queue "Zoom", open the generated keyword queue "Zoom", and open the zoom function Associate with the touch control dialog. In embodiments, the interface analyzer may refer to a predetermined or custom keyword cue library when associating the touch control dialog with the keyword cue, as described in more detail with reference to FIG. 7B.

At block 422, the display displays a first instance of the user interface. In an embodiment, the headset engine 210 adjusts the display of the user interface as described with reference to FIG. In an embodiment, at block 424, a graphical overlay including at least one visual indicator of a keyword cue identified in the first user interface to the interface analyzer may be simultaneously displayed on the first user interface. In a further embodiment, the visual indicia may be located substantially proximate to the location of the touch control dialog, as shown and described with reference to the figures.

  At block 426, an audible input is detected by the headset. In embodiments, the audible input, for example, the phrase “zoom” spoken by the user, may be first detected by the voice input 224. The audio input 224 may then convert the spoken phrase “zoom” into an audio feed and pass the audio feed to the audio detection component 252. The audio detection component 252 can then invoke the audio processing component 254.

  At block 428, the audio input is processed and analyzed to determine whether the audible input matches the keyword cue, and thus the touch control dialog. In embodiments, the speech processing component 254 may use speech recognition algorithms, noise reduction algorithms, speech-to-text algorithms, machine learning algorithms, etc. to process the speech feed. For example, the audio feed may be processed to separate the phrase "zoom" spoken by the user from ambient noise, incidental noise, or background noise. The audio processing component 254 can then analyze the processed audio feed and compare the processed audio feed to a keyword queue to determine whether the processed audio feed matches the keyword queue.

  At block 430, a touch-sensitive command corresponding to the touch-sensitive control dialog associated with the detected keyword cue is generated, and the command is executed within the first instance of the touch-sensitive user interface. In other words, if the audio processing component determines that the audio feed matches the keyword cue, the audio conversion component converts the matched keyword cue to an associated control dialog. Next, the voice conversion component passes the control dialog to the generation command generation unit a command equivalent to the touch type control dialog. The generated command is then executed by the headset engine in the touch-based user interface. For example, if the audio processing component matches the phrase "zoom" with the keyword queue "zoom", the audio conversion component converts the keyword queue "zoom" into a command equivalent to selecting a zoom control dialog. Thereafter, the command is passed to the command generator, and the command generator generates a command equivalent to the touch-type user selection of the zoom touch-type control dialog. Next, the command is executed by the headset engine in the touch-based user interface to enable the zoom function. It will be appreciated that the method 416 can be performed iteratively as many times as necessary.

  Referring now to FIG. 4c, a block diagram is provided illustrating an exemplary motion-based and audio-based hands-free interaction mode that may be implemented at least partially by the headset shown in FIG. At block 440, a hands-free interaction mode is initiated. Analyze the touch-based OS to detect at least one touch-based command associated with the first instance of the user interface, as described with reference to FIG. For example, the interface analyzer may detect that the first instance of the user interface, for example for navigating to a different menu screen of the touch operating system, comprises a touch scroll function, and the interface analyzer 212 may include: The first instance of the user interface may, for example, detect that it comprises a touch control dialog for opening an application installed on the touch operating system.

  Next, the headset engine 210 can invoke the hands-free movement protocol 436 and activate the hands-free voice protocol 438. In embodiments, the hands-free movement protocol 436 may comprise some or all of the process of the motion-based hands-free interaction mode 400. For example, the headset engine 210 can invoke the calibration control component 246 to determine an initial position of the head mounted computing device, including the orientation of the head mounted computing device with respect to one or more axes. In embodiments, hands-free voice protocol 438 may comprise some, all, or an alternative process of voice-based hands-free interaction mode 416. For example, the headset engine can invoke a control dialog detection module to enable audio hands-free navigation. It will be appreciated that the headset engine may initiate the hands-free movement protocol 436 and the hands-free voice protocol 438 in any order or simultaneously.

  Once the initial reference heading of the headset has been determined and the touch control dialog has been associated with the keyword cue, a first instance of the touch user interface is displayed, as shown in block 440. At block 442, the motion detection component and the audio detection component monitor, detect, and analyze input data from the headset I / O system as described with reference to FIGS. 4A and 4B.

  At block 444, when a movement is detected, the motion detection component determines whether the movement has exceeded an associated threshold, as described with reference to block 410. If the associated threshold is exceeded, the display is adjusted at block 446, as described at blocks 412 and 414. If the associated threshold has not been exceeded, the system returns to block 442.

  At block 448, if a speech input is detected, the speech processing component determines whether the speech input matches the keyword cue, as described with reference to block 428. If the voice input matches the keyword cue, a touch-sensitive command equivalent to the associated control dialog is executed at block 450, as described with reference to block 430. It will be appreciated that the method 432 can be performed iteratively as many times as necessary.

  Referring now to FIG. 5, a block diagram is provided illustrating an exemplary multi-axis motion-based method 500 for facilitating hands-free interaction with a touch-based operating system. In other words, the method 500, when utilized with a compatible headset, facilitates a simple and complex motion-based hands-free interaction with a touch-based operating system. As an illustrative example, at block 510, a user wearing a compatible headset initiates a hands-free navigation interface. As described above, the initial reference orientation of the headset is determined to be equivalent to pointing forward at the neutral position. At block 512, the headset detects movement. For example, the user turns his head from neutral to right, and simultaneously up. At block 514, the system determines whether the movement is within the first axis. For example, the movement in the first axis can be a rotation about the z-axis from the initial reference position toward the x-axis (to the right of the user). In other words, the system can detect that the user has turned (turned) his head from the front to the right. Also, at block 516, the system determines whether the movement has occurred on the second axis. For example, the movement in the second axis may be a rotation about the x-axis from the initial reference position toward the z-axis (upward from the wearer's point of view). In other words, the system can detect that the user has turned (rotated) his head from a position substantially parallel to the y-axis to the z-axis.

  At block 518, the system determines whether the movement in the first axis has exceeded a first axis threshold. The first axis threshold value may be determined in advance to be, for example, ± 10 degrees from the initial reference position. If the system detects a displacement in the first axis that exceeds ± 10 degrees, the system determines that the threshold has been exceeded. If the system detects a displacement of ± 10 degrees or less on the first axis, the system determines that the threshold has not been exceeded. For example, if the user turns his head 20 degrees to the right, the system will determine that the first axis threshold has been exceeded.

  At block 520, the system determines whether the movement in the second axis has exceeded a second axis threshold. The second axis threshold value may be predetermined so as to be, for example, ± 10 degrees from the initial reference position. If the system detects a displacement in the second axis that exceeds ± 10 degrees, the system determines that the threshold has been exceeded. If the system detects a displacement of the second axis of less than ± 10 degrees, the system determines that the threshold has not been exceeded. For example, if the user turns his head up 20 degrees, the system will determine that the second axis threshold has been exceeded.

  At block 522, the system performs a touch scroll function associated with the multi-axis input and displays a second instance of the touch user interface. In embodiments, the multi-axis input may represent two independent touch scroll functions. Continuing with the previous example, multi-axis input (right and up) involves moving the horizontal scroll bar to the right by a predetermined number of scroll units and moving the vertical scroll bar up by a predetermined number of scroll units. Can respond. In embodiments, the multi-axis input may represent a single touch scroll function. For example, multi-axis input (upper right) may correspond to moving the vertical scroll bar by a number of scroll units corresponding to all pages of the displayed document.

  Returning to block 516, if movement is detected only on the first axis, the system determines at block 524 whether the movement has exceeded the first axis threshold. If the movement has not exceeded the first axis threshold, the system returns to block 512. If the movement exceeds the first axis threshold, at block 526, the system performs a touch scroll function associated with the first axis input and displays a second instance of the touch user interface.

  Returning to block 518, if the detected movement does not exceed the first axis threshold, the system proceeds to block 528. At block 528, the system determines whether the movement has exceeded a second axis threshold. If the movement has not exceeded the second axis threshold, the system returns to block 512. If the movement exceeds the second axis threshold, at block 530, the system performs a touch scroll function associated with the second axis input and displays a second instance of the touch user interface.

  The examples provided with respect to the example method 500 represent only a subset of the possible multi-axis inputs and associated touch scroll commands within the scope of the present disclosure, and these examples are merely illustrative and are to be construed as limiting. It should be understood that this is not intended.

  Referring now to FIG. 6A, an exemplary diagram is provided illustrating possible use cases consistent with embodiments of the disclosed invention. FIG. 6A includes only a subset of the systems and components of the embodiments for clarity of the illustration. After initiating the hands-free navigation interface, a user wearing a compatible headset is presented, via a display, with a first portion of a blueprint 602 relating to the building the user is building. . After inspecting the area of the building 600 associated with the first portion of the blueprint 602, the user can turn his head to the left. At 604, the system detects movement in a first axis, in this case, the x-axis, and determines that movement in the first axis has exceeded a first axis threshold. The system then translates the movement into a command related to a touch-based scrolling function equivalent to scrolling to the left. Thereafter, the system executes the command within the touch-based user interface. Next, the system displays a second instance of the touch-based user interface. Here, the command causes the displayed blueprint to scroll left by a predetermined distance, and the display shows the second portion of blueprint 606.

  Referring now to FIG. 6B, an exemplary diagram illustrating another possible use case consistent with embodiments of the disclosed invention is provided. FIG. 6B includes only a subset of the systems and components of the embodiments for clarity of the illustration. After launching the hands-free navigation interface and detecting a vertical scroll bar at scroll bar position 616a and a horizontal scroll bar at scroll bar position 614a, the user is prompted through a display to a user wearing a compatible headset. The first page of expense report 612 relating to the building in question is displayed. After reviewing the first page of expense report 612, the user can rotate his head down. At 618, the headset may detect angular displacement about the x-axis and determine that motion in the first axis exceeds a first axis threshold. The system then translates the movement into a command associated with moving the scrollbar position 616a downward a predetermined distance from the scrollbar position 616b. Thereafter, the system executes the command within the touch-based user interface. Next, the system displays a second instance of the touch-based user interface. Here, the display displays the second page of the expense report 620.

  Referring now to FIG. 7A, a flow diagram is provided illustrating a method 700 that generally facilitates accurate conversion of speech input into commands executed within a touch-based user interface of a touch-based operating system. First, as shown at block 701, the method includes receiving input data from a headset sensor. Further, at block 702, the method may include determining whether the received input data is an audio feed. For example, a voice input associated with a touch control dialog may be detected. In some aspects, at block 704, the method includes analyzing the audio input. In embodiments, the audio feed is processed using computer-based speech recognition techniques to identify audio input. At block 706, the method may determine whether the analyzed speech input matches a keyword cue. In an embodiment, the analyzed speech input is compared to a generated keyword cue associated with the touch-based control dialog by the control dialog detector. It should be understood that one or more speech-text matching algorithms can be used to determine whether a speech input matches a keyword cue. In embodiments, the touch control dialog may be associated with a mobile application running on a touch operating system. In an embodiment, at block 708, the method includes determining whether a keyword cue that matches the voice input corresponds to an instruction to suspend the hands-free interaction mode. For example, the method may determine whether the keyword cue corresponds to an instruction to set the voice input component to passive mode. In this context, passive mode refers to pausing (at least temporarily) analyzing a headset sensor or voice input system within a hands-free interactive mode. However, in embodiments, the headset sensor and / or the voice input system remain active for use in other processes. That is, the headset sensor and / or voice input system may continue to transmit data to applications running within the touch-based operating system. For example, after responding to a video chat request from a video chat application using the hands-free interaction mode, the user can put the hands-free interaction mode into the passive mode by speaking the keyword queue associated with the passive mode. However, the use of headset microphones and cameras in video chat applications will continue.

  In an embodiment of the method, at block 710, the voice input is set to passive mode in response to determining that the keyword cue is associated with a command to set the voice input to passive mode. In some embodiments, at block 712, a command to exit passive mode is received, and the method returns to block 702. In embodiments, the command to exit passive mode may be associated with a physical input element (such as a button) located on the headset. Returning to block 708, in embodiments, if the keyword cue is other than a passive mode command, a touch control dialog command is generated and executed within the touch user interface.

  Referring now to FIG. 7B, a portion of the flow diagram shown in FIG. 7A is provided that illustrates an exemplary method 716 for comparing speech input to a keyword cue. The example method 716 generally facilitates comparing audio input with the generated keyword cues, custom keyword cues, and auxiliary keyword cues. As described with reference to FIG. 7A, at block 704, the method includes analyzing the audio input. At block 718, the analyzed speech input may be compared to a keyword cue generated by the interface analyzer. As described above, the keyword queue may be identified and generated by the interface analyzer by analyzing the source code associated with the first instance of the user interface.

  In some embodiments, the analyzed speech input may be compared to a custom keyword cue library 722 at block 720. Thus, a user can create a customized keyword cue and associate the customized keyword cue with the touch control dialog. In embodiments, the custom keyword library may at least partially replace a keyword cue generated by the interface analyzer. In embodiments, the keyword queue may be a combination of customized or predetermined keyword queue control dialogs associated with a particular first instance of the user interface.

  In an embodiment, at block 724, the analyzed speech input may be compared to an auxiliary keyword cue library 728. In embodiments, the auxiliary keyword cue library may include a table containing a plurality of keyword cues associated with the touch control dialog. For example, if the interface analyzer identifies a touch control dialog that cannot be pronounced, the interface analyzer may automatically replace at least one auxiliary keyword cue from the auxiliary keyword cue library associated with the non-pronounceable control dialog. Additionally and / or alternatively, if the first instance of the touch-based user interface includes a plurality of touch-based control dialogs that result in substantially similar generated keyword cues, the auxiliary keyword cue library may be similar. May be provided with an alternative keyword queue.

  Referring now to FIG. 8A, an exemplary touch-based user interface showing an exemplary instance of a detected control dialog is provided. That is, 1 to 28 indicate interface functions identified as touch-type control dialogs. In embodiments, the interface analyzer 212 may detect the interface functions 1-28 by analyzing source code associated with the user interface. In the depicted example, interface analyzer 212 may analyze source code associated with an application running within the touch-based operating system currently presented by the touch-based user interface. However, the interface analyzer 212 may also analyze source code or other existing code associated with the touch-based operating system itself.

  In an embodiment, when an interface function (e.g., interface functions 1-28) is identified as a touch control dialog by the interface analyzer 212, the interface analyzer 212 analyzes the control dialog, generates a keyword queue, and creates a keyword queue. Can be associated with a control dialog. In embodiments, the interface analyzer 212 may re-analyze the relevant source code. In embodiments, interface analyzer 212 may provide at least a partial analysis of the relevant source code to interface analyzer 212.

  Referring now to FIG. 8b, an exemplary touch-based user interface having a keyword cue overlay is shown. In other words, 30-38 show display overlays that may be presented to a user to assist in using audio hands-free navigation according to embodiments of the present disclosure. In embodiments, the overlay may be automatically displayed by the headset engine 210 based on an analysis of the touch-based user interface by the interface analyzer 212. In embodiments, the overlay may be automatically generated by multiple sub-components of headset engine 210. For example, the interface analyzer 212 can detect an interface function (such as the interface functions 1-28 in FIG. 8A), identify the interface function as a control dialog, and trigger the voice detection component 252. The voice detection component 252 analyzes the control dialog, generates a keyword cue, associates the keyword cue with the control dialog, the headset engine 210 detects the association, generates an overlay including the keyword cue, and touches the overlay to a touch-based user. It may be overlaid on the display of the interface (such as overlays 30-38). It should be understood that the above is used only as an exemplary method for creating an overlay consistent with the present disclosure and is not meant to be limiting.

  However, the automatic creation of the overlay may cause more control dialogs to be displayed than necessary, useful, or desirable to be displayed on the overlay for a particular instance of the touch-based user interface. In other words, purely automatic generation of keyword cue overlays can inadvertently impede hands-free navigation of the touch-based user interface. Thus, in embodiments, the overlay may be automatically displayed by the headset engine 210 based on the customized preferences. In such embodiments, headset engine 210 may identify a predetermined overlay template associated with an instance of the touch-based user interface in a custom library.

  In embodiments, the functionality of the control dialog may be determined by the headset engine 210 and / or its sub-components, and the overlay may be generated only for relevant keyword cues determined to be relevant to the user. In the embodiment, this determination may be made by the environment analysis unit 214 based on the environment data. In embodiments, the determination may be made based at least in part on user preferences (eg, user preferences stored in user data 274).

  Referring now to FIG. 9, an exemplary head mounted computing device 900 consistent with some embodiments of the present disclosure is shown. Head-mounted computing device 900 generally facilitates hands-free interaction with a touch-based user interface of a touch-based operating system. Although the exemplary head-mounted computing device 900 is shown with various sensors, it will be appreciated that the locations and numbers of the sensors may vary in embodiments without departing from the scope of the present disclosure. In an embodiment, the head-mounted computing device 900 may include multiple sensors for sensing motion and sound, and components for displaying a touch-based user interface to a user. For example, a typical display 902 generally facilitates displaying a touch-based user interface to a user. In an embodiment, display 902 may be configured with head-mounted computing device 900 such that display 902 is present for displaying a touch-based user interface. In embodiments, display 902 may be further configured to display a keyword cue overlay, as described above. In embodiments, the display 902 may be at least partially translucent, allowing a user to view the display 902 and perceive both the displayed touch-based user interface and the environment. In embodiments, display 902 may be a monocular display. In embodiments, display 902 may be a binocular display. However, it will be understood that any display may be used in embodiments of the disclosed invention without departing from the scope of the present disclosure. In embodiments, the display 902 may be configured to communicate with at least the headset device I / O 222.

  The exemplary sensor 904 generally facilitates capturing audio input. In embodiments, sensor 904 may be a directional microphone. In embodiments, sensor 904 may be an omni-directional microphone. In an embodiment not shown, the sensor 904 is a plurality of microphones configured to capture and ultimately use ambient noise to assist in processing and analyzing user voice input, including head-mounted computing It may further include a plurality of microphones located at various points on the device 900. It will be appreciated that the sensor 904 can be any sensor or system of sensors that can sense an audio input and convert the audio input into an audio feed without departing from the scope of the present disclosure. In embodiments, the example sensor 904 may be configured to communicate with the hands-free input determiner 240 and its subsystems.

  The example physical input sensor 906 generally provides an input component that facilitates restarting various hands-free navigation features. In embodiments, the physical input sensor 906 may be configured to send a signal to the hands-free input determiner 240 to restart the voice command after the passive command has been issued, as described with respect to FIG. 7A. . In embodiments, the physical input 906 may be configured to send a signal to the hands-free input determiner 240 to recalibrate the headset reference orientation as described above. In an embodiment, the physical input sensor 906 may include a plurality of physical input sensors, wherein the first sensor is configured to send a signal to the hands-free input determination unit 240 to restart the voice command, Two sensors may be configured to send a signal to the hands-free input determiner 240 to recalibrate the reference orientation. It will be appreciated that the physical input sensor 906 can be any sensor that can detect a physical interaction without departing from the scope of the present disclosure.

  The example audio output 908 generally provides audio output from the head mounted computing device 900 to a user. It is to be understood that embodiments may use any audio output component (such as a speaker) that can generate sound in response to an electrical input without departing from the scope of the present disclosure. In embodiments, the audio output 902 may be configured to communicate with at least the headset device I / O 222.

  The exemplary motion sensor 910 generally facilitates motion detection of the motion processing system described above. As used herein, a motion sensor may include at least one accelerometer, multi-axis accelerometer, magnetometer, gyroscope, capacitive transducer, potentiometer, resistive transducer, synchro, or motion in at least one axis. Any similar sensor that can detect the In embodiments, the motion sensor 910 may include at least one motion sensor. In an embodiment, the motion sensor 910 may include a plurality of motion sensors. In embodiments, motion sensor 910 may be configured to communicate with hands-free input determiner 240 and its subsystems.

  In embodiments, the example sensor 912 generally makes a motion determination. In embodiments, the sensor 912 may be a light-sensitive digital sensor configured to periodically capture images (such as at 60 frames / second or any predetermined rate). In embodiments, the image capture component 912 may be a light sensitive digital sensor configured to capture images continuously. In embodiments, sensor 912 may be configured to communicate with hands-free input determiner 240 and its subsystems. It is understood that the exemplary sensor 912 can include any sensor that can capture digital images (cameras, video cameras, etc.) that can be used in embodiments without departing from the scope of the present disclosure. Would.

  As referred to herein, displacement may refer to any change in the position of a headset (such as head-mounted computing device 900) in nine degrees of freedom with respect to three axes. This includes, but is not limited to, translation or rotation in any of the three axes. Note that the terms used in relation to displacement in three-dimensional space vary widely from field to field (aviation, biomechanics, computer science, etc.) and may vary widely in common usage. For this reason, every attempt has been made to clarify and simplify the description that describes movement, displacement, rotation, and / or angular displacement. However, unless expressly stated to the contrary, each example provides a context and is not limiting of the disclosure.

  For example, a translation of the x-axis can be said to move right or left. However, it should be understood that this can be considered equivalent to moving from the origin on the horizontal axis (positive (right) or negative (left)). Rotation about the x-axis (angular displacement) can be said to rotate up or down. However, it should be understood that this can be considered equivalent to pitch up or pitch down. Therefore, it can be understood that, for example, moving to the right while rotating upward moves from the origin toward a positive value on the horizontal axis while increasing the pitch.

  Translation of the y-axis can be said to translate forward or backward. However, it should be understood that this can be considered equivalent to a movement from the origin on the vertical axis (positive (forward) or negative (back)). Rotation (angular displacement) about the y-axis can be said to tilt left or right. However, it should be understood that this can be considered equivalent to rotating left (counterclockwise) or right (clockwise). Thus, for example, when moving forward while tilting to the left, it can be understood that moving from the origin toward a positive value on the vertical axis while rotating to the left.

  It can be said that the parallel movement of the z axis moves up and down. However, it should be understood that this can be considered equivalent to moving from the origin of the vertical axis (positive (up) or negative (down)). Rotation (angular displacement) about the z-axis can be referred to as left or right rotation. However, it should be understood that this can be considered equivalent to waving left or right. Thus, for example, moving up while rotating to the left may be understood as moving from the origin toward a positive value on the vertical axis while swinging to the left.

  Having described various embodiments of the present disclosure, an exemplary computing environment suitable for implementing the embodiments of the present disclosure will now be described. Referring to FIG. 10, an exemplary computing device is provided and is generally referred to as a computing device 1000. Computing device 1000 is merely an example of a suitable computing environment and is not intended to imply limitations as to the scope of use or functionality of the present disclosure. Neither should the computing device 1000 be interpreted as having any dependency or requirement relating to any one or combination of the illustrated components.

  Embodiments of the present disclosure are directed to computer code or a machine that includes computer-usable or computer-executable instructions, such as program modules, that are executed by a computer or other machine, such as a personal data assistant, smartphone, tablet PC, or other form device. This can be explained in the general context of the enabling directive. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. Embodiments of the present disclosure may be implemented in various system configurations, including handheld devices, consumer electronics, general purpose computers, more specialized computing devices, and the like. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located on both local and remote computer storage media, including memory storage devices.

  Referring to FIG. 10, a computing device 1000 includes a bus 1010 that directly or indirectly couples the following devices. That is, bus 1010 includes memory 1012, one or more processors 1014, one or more display components 1016, one or more input / output (I / O) ports 1018, one or more I / O components. 1020 and the exemplary power supply 1022. Bus 1010 represents what may be one or more buses (such as an address bus, a data bus, or a combination thereof). Although the various blocks in FIG. 10 are shown by lines for clarity, in reality these blocks are not logical, but necessarily actual components. For example, a display component such as a display device may be considered an I / O component. Also, because the processor comprises a memory, the memory 1012 and the one or more processors 1014 may or may not include separate or distributed components. The inventor of the present invention recognizes that it is a property of the art, and the diagram of FIG. 10 is merely an example of an exemplary computing device that may be used in connection with one or more embodiments. He reiterates a certain thing. No distinction is made between categories such as “workstation”, “server”, “laptop”, “handheld device”, etc., all within the scope of FIG. 10 and with reference to “computing devices”.

  Computing device 1000 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computing device 1000 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technique for storing information such as computer readable instructions, data structures, program modules or other data. Both are included. Computer storage media includes RAM, ROMEEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, magnetic storage, or Includes, but is not limited to, any other storage media that can be used to record the desired information and accessible by the computing device 1000. A computer storage medium does not itself contain signals. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above also fall within the scope of computer readable media.

  Memory 1012 includes computer storage media in the form of volatile and / or non-volatile memory. The memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard drives, optical disk drives, and the like. Computing device 1000 includes one or more processors 1014 that read data from various entities, such as memory 1012 or I / O components 1020. Display component 1016 presents the data display to a user or other device. Exemplary display components include display devices, speakers, printing components, vibration components, and the like.

  I / O ports 1018 allow computing device 1000 to be logically coupled to, and may incorporate portions of, other devices, including I / O components 1020. Exemplary components include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, and the like. The I / O component 1020 may provide a natural user interface (NUI) for processing air gestures, voice, or other physiological input generated by a user. In some cases, the input is sent to the appropriate network element for further processing. The NUI includes voice recognition, touch and stylus recognition, face recognition, biometrics, air gestures on the screen and adjacent to the screen, gesture recognition by head and eye tracking, and touches associated with the display of the computing device 1000. Any combination of recognition may be implemented. Computing device 1000 may include a depth camera, such as a stereoscopic camera system, an infrared camera system, an RGB camera system, and combinations thereof, for gesture detection and recognition. In addition, computing device 600 may include an accelerometer or gyroscope that allows for the detection of motion. The output of the accelerometer or gyroscope may be provided on a display of the computing device 1000 to depict immersive augmented or virtual reality.

  Some embodiments of the computing device 1000 may include one or more radios 1024 (or similar wireless communication components). The wireless device 1024 transmits and receives wireless telecommunications or wireless communication. Computing device 1000 may be a wireless terminal adapted to receive communications and media over various wireless networks. Computing device 1000 may communicate with other devices, such as code division multiple access (“CDMA”), a global system for mobile (“GSM®”), or time division multiple access (“TDMA”). It can communicate via wireless protocols. The wireless communication may be a short-range connection, a long-range connection, or a combination of both short-range and long-range wireless communication connections. References to "short" and "long" type connections do not refer to the spatial relationship between the two devices. Instead, short-range and long-range are generally referred to as different categories or types of connections (ie, primary and secondary connections). A short-range connection includes, by way of example and not limitation, a Wi-Fi connection to a device (such as a mobile hotspot) that provides access to a wireless communication network, such as a WLAN connection using the 802.11 protocol. included. A Bluetooth® connection to another computing device is a second example of a short-range or short-range communication connection. Long distance connections may include, by way of example and not limitation, connections that use one or more of the CDMA, GPRS, GSM, TDMA, and 802.16 protocols.

Many different arrangements of the various components shown are possible, as are the components not shown, without departing from the scope of the claims. The embodiments of the present disclosure have been described for purposes of illustration and not limitation. Alternative embodiments will become apparent to the reader of the present disclosure after and after reading this specification. Alternatives for implementing the foregoing may be envisaged without departing from the scope of the claims. Certain features and sub-combinations are useful and can be used without reference to other features and sub-combinations and are contemplated within the scope of the claims.

Claims (20)

  A computer-implemented method for hands-free navigation of a head-mounted computing device, wherein initiating a hands-free interaction mode on the head-mounted computing device that enables interaction with a user interface of a touch-based operating system. Detecting an initial position of the head-mounted computing device including a position of the head-mounted computing device with respect to one or more axes; and displaying the touch-based operating system on a display of the head-mounted computing device. Displaying a first instance of a user interface; and detecting a first displacement of the head mounted computing device. Step, wherein the first displacement is a displacement that exceeds a first threshold displacement that is a displacement with respect to the initial position, and one or more touch-based inputs based on the first displacement. A method comprising: determining a corresponding first command; and communicating the first command to the touch-based operating system.   The method of claim 1, further comprising: receiving a first audible input associated with the user interface of the touch-based operating system; and determining a second command based on the first audible input. Computer implementation method.   The computer-implemented method of claim 1, wherein the user interface is associated with an electronic document on the touch-based operating system.   The computer-implemented method of claim 2, wherein the first command comprises a scroll command having a fixed number of scroll units.   The computer-implemented method of claim 1, wherein the first threshold displacement corresponds to a minimum angular displacement of the head mounted computing device with respect to the one or more axes.   The computer-implemented method of claim 1, wherein the first displacement is detected by one or more sensors of the head mounted computing device.   The computer-implemented method of claim 1, wherein detecting the first displacement comprises determining a duration of the first displacement.   The computer-implemented method of claim 1, further comprising displaying a second instance of the user interface of the touch-based operating system on a display of the head-mounted computing device.   Detecting a second displacement of the head mounted computing device, wherein the second displacement is a displacement that exceeds a second threshold displacement, and one based on the second displacement. Or determining a second command corresponding to the plurality of touch-based inputs and displaying a third instance of the user interface of the touch-based operating system on the display of the head-mounted computing device. A computer-implemented method according to claim 8.   Hands used by one or more computing devices to enable the one or more computing devices to interact with a touch-based operating system user interface on the head-mounted computing device Initiating a free interactive mode; identifying at least one touch control dialog in the user interface of the touch operating system; and displaying the user interface of the touch operating system on a display of the head mounted computing device. Displaying a first instance of a first audible input associated with the at least one touch control dialog. Attaching to the head-mounted computing device in response to the first audible input and displaying a second instance of the user interface of the touch-based operating system. A non-transitory computer storage media device that stores instructions.   The computer storage medium of claim 10, further comprising displaying at least one graphical overlay control on a display of the head mounted computing device over the first instance of the user interface.   The computer storage medium of claim 11, wherein the at least one graphical overlay control comprises a visual indicator representing an audible input corresponding to the at least one touch control dialog.   The method of claim 10, further comprising selecting the at least one graphical overlay control based on an identification of a graphical overlay control of the plurality of predetermined graphical overlay controls associated with the at least one touch control dialog. Computer storage media.   The computer storage medium of claim 10, further comprising receiving a second audible input corresponding to a command to suspend the hands-free interactive mode.   The computer storage medium of claim 10, further comprising performing speech recognition to recognize text from the first audible input.   The computer storage medium of claim 10, wherein the user interface is associated with a mobile application running on the touch-based operating system.   A system for hands-free navigation of a head-mounted computing device, comprising: one or more processors; and one or more computer storage media, wherein the one or more computer storage media comprises: Initiating a hands-free interaction mode that is executed by one or more processors to enable interaction with a touch-based operating system user interface on the head-mounted computing device; Displaying a first instance of the user interface of the touch-based operating system on a display of a touching device; and a threshold displacement of the head mounted computing device. Detecting a first hands-free input comprising one or more of a first displacement exceeding and a first audible input associated with a touch-based control dialog included in the first instance of the user interface. And determining a first command corresponding to one or more touch-based inputs based on the hands-free input; and displaying the user of the touch-based operating system on the display of the head-mounted computing device. Displaying a second instance of the interface; storing computer-usable instructions that direct the method.   The system of claim 17, wherein the first hands-free input includes the first displacement and the first audible input.   The implemented method comprises: detecting a second hands-free input; and determining a second command corresponding to one or more touch-based inputs based on the second hands-free input. 18. The system of claim 17, further comprising: displaying a third instance of the user interface of the touch-based operating system on the display of the head-mounted computing device. The method of claim 17, wherein the implemented method further comprises: detecting a hands-free start input; and generating a command to start the hands-free interactive mode on the head-mounted computing device. system.

Images