CN102339129B - Multichannel human-computer interaction method based on voice and gestures - Google Patents

Abstract

The invention discloses a multi-channel human-computer interaction method based on voice and gesture, which extracts voice reference object constraint information from voice information, and extracts gesture reference object constraint information from gesture information, wherein the gesture reference object constraint information includes current pointing The statistics of the distance from any point in the pointing area defined by the gesture to the pointing center of the pointing gesture and the time statistic maintained by the pointing gesture. When analyzing the constraint information of the gesture reference object, the distance statistics and time statistics are obtained to reduce the ambiguity of pointing in 3D interaction. In the process of determining the referent object, the model objects in the virtual environment are divided into four categories, and the referent object is compared with a certain type of model object according to the possibility of the referent object appearing in a certain type. It is also helpful to narrow the search scope of the referring object and reduce the impact of pointing ambiguity.

Description

A Multi-channel Human-Computer Interaction Method Based on Speech and Gesture

technical field

The invention relates to the field of human-computer interaction, in particular to a multi-channel human-computer interaction method based on voice and gesture.

Background technique

Multi-channel human-computer interaction can effectively expand the bandwidth of information exchange between human and computer, so as to achieve the purpose of improving interaction efficiency; it can also exert the different cognitive potentials between human and computer, and reduce the cognitive load of users. Users can complete interactive tasks through various interaction channels and their mutual combination and cooperation, which just makes up for the limitations and burdens brought to users by a single interaction mode. In multi-channel human-computer interaction, referential resolution is defined as finding out the common referent of input information from multiple channels. Among them, reference mainly includes pronouns, positioning adverbs, demonstratives and qualifiers in natural language, such as "it", "here", "this", "that house" and so on; the object of reference is the objective entity referred to by the user, For example, models in three-dimensional space, etc. In traditional single-channel user interfaces, the referential technique is singular and often precise, and the boundaries between objects and objects are clear. In MUI, however, referential techniques are compounded and often blurred, and boundaries are not clear.

At present, multi-channel research is not limited to the integration of voice and traditional mouse and keyboard. Multi-channel systems based on voice and pen, voice and lip movement, voice and three-dimensional gestures have received greater attention. Typical representatives include QuickSet, a multi-channel collaboration system based on Agent structure that supports voice and pen, and XWand system that integrates "magic wand" (a new six-degree-of-freedom device) and voice. The W3C international organization has established a "multi-channel interaction" working group to develop a new W3C multi-channel protocol standard supporting mobile devices, including the multi-channel interaction framework, multi-channel interaction requirements, multi-channel interaction use cases, and scalable multi-channel annotations Language requirements, digital ink requirements, Extensible Multi-Channel Annotation Markup Language, etc. The establishment of these standards reflects the maturity of multi-channel technology.

Regarding the research on the problem of referential resolution in multi-channel human-computer interaction, Kehler used the relevant principles of cognitive science and computational linguistics to study and verify the corresponding relationship between referential and cognitive states in a multi-channel environment, and proposed a method for understanding cognitive states. The method of encoding and combining a set of simple judgment rules to obtain the referring object has achieved high accuracy in a pen and voice-based 2D tourist map application. Kehler's method works well when dealing with single-pointing gestures combined with precise pointing gestures, but these rules assume that all objects can be deterministically selected and cannot support ambiguous gestures.

Columbia University, Oregon University of Science and Health, etc. collaborated to study three-dimensional multi-channel interaction in augmented reality and virtual reality environments, and proposed to use the method of perceiving shapes to solve the problem of referential resolution. Perceived shapes are user-controlled geometries through which users interact with augmented reality or virtual reality environments. Perceived shapes generate various statistical information during the interaction process to assist target selection. This method mainly solves the problem of pointing ambiguity in referential resolution, but does not pay attention to the inference of unspecified information and multi-channel alignment. Pfeiffer et al. from Bielefeld University in Germany proposed that multi-channel referential resolution should pay attention to referential types, sentence complexity, consistent background, uncertainty, etc., and designed a referential resolution engine for immersive virtual environments. The engine is an expert system with three layers: core layer, domain layer and application layer. The core layer is a constraint satisfaction manager; the domain layer provides access to the knowledge base; the application layer is the interface between external programs and the reference resolution engine, and is responsible for converting references in speech input into queries for the reference resolution engine. The reference resolution engine regards the reference resolution problem as a constraint satisfaction problem, and mainly focuses on extracting effective constraints from complex natural language. However, this method still lacks corresponding treatment for under-constrained situations and pointing ambiguity.

Contents of the invention

The invention designs and develops a multi-channel human-computer interaction method based on voice and gesture.

An object of the present invention is to solve the pointing ambiguity problem in the multi-channel human-computer interaction method based on voice and gesture. When performing three-dimensional interaction in a virtual environment, gestures (from the beginning of pointing recognition to the end of pointing) not only express spatial information, but also carry temporal information. The longer the object stays in the pointing area, the more likely it is to be selected. Therefore, when analyzing the constraint information of gesture referent objects, not only distance statistics but also time statistics should be obtained, so as to reduce pointing ambiguity in 3D interaction. Moreover, in the process of determining the referent object, the model objects in the virtual environment are divided into four categories, and the referent object is compared with a certain type of model object. This method also helps to narrow the search scope of the referent object , reducing the effect of pointing ambiguity.

Another object of the present invention is to solve the problem of unspecified information inference in the multi-channel human-computer interaction method based on voice and gesture. The model objects in the virtual environment are divided into four categories. Among them, the focus object is the referring object determined during the last human-computer interaction, that is, if there is The indicative pronoun "it" can be considered as the referring object of this human-computer interaction is the focused object, thus solving the problem of unspecified information inference.

Another object of the present invention is to provide a multi-channel human-computer interaction method based on speech and gestures. By building a multi-channel hierarchical integration model, four layers are established in the multi-channel hierarchical integration model: physical layer, lexical layer, grammatical layer and semantic layer, and finally the command information and referent objects required for human-computer interaction are filled into tasks The goal of the above integration process and the criteria for the success of the integration are based on the integrity of the human-computer interaction task structure. The ultimate goal is to generate a task structure that can be submitted to the system for execution to ensure the effective implementation of human-computer interaction.

The technical scheme provided by the invention is:

A multi-channel human-computer interaction method based on voice and gesture, is characterized in that, comprises the following steps:

Step 1. Build a voice channel and a gesture channel, and input voice information and gesture information to the referent of the human-computer interaction through the voice channel and the gesture channel respectively;

Step 2. Extract the speech reference object constraint information from the above voice information, and extract the gesture reference object constraint information from the above gesture information, wherein the gesture reference object constraint information includes the arrival of any point in the pointing area defined by the current pointing gesture The distance statistics of the pointing center of the pointing gesture and the time statistics maintained by the above-mentioned pointing gesture;

Step 3: Compare the above-mentioned speech referent object constraint information and gesture referent object constraint information with the feature information of the model object in the virtual environment, determine the referent object of human-computer interaction, and extract the referent object from the above-mentioned speech referent object constraint information Command information, which acts on the referent object to complete a human-computer interaction.

Preferably, in the multi-channel human-computer interaction method based on voice and gestures, the model objects in the virtual environment are divided into four categories: pointing objects, focusing objects, activation objects and silent objects, and the pointing objects are The object located in the pointing area defined by the current pointing gesture, the focus object is the referent object determined during the last human-computer interaction, and the activation object is the pointing object and activation object located within the visible range The silent object is a model object other than the pointing object and the activation object located in the invisible range. In step 3, the above-mentioned voice referent object constraint information and gesture referent object constraint information are sequentially combined with the above-mentioned The feature information of the pointing object, focusing object, active object, and silent object is compared to determine the referent object of human-computer interaction.

Preferably, in the multi-channel human-computer interaction method based on voice and gesture, in the second step,

Extracting the speech reference object constraint information from the above voice information and the gesture reference object constraint information from the above gesture information are realized in the following ways:

Build a multi-channel layered integration model, the multi-channel layered integration model includes four layers, respectively physical layer, lexical layer, syntax layer and semantic layer, wherein, the physical layer receives input from voice channel and gesture channel respectively speech information and gesture information, the lexical layer includes a speech recognition and analysis module and a gesture recognition and analysis module, the speech recognition and analysis module parses the speech information of the physical layer into speech reference object constraint information, and the gesture recognition and analysis module will The gesture information at the physical layer is parsed into constraint information of the gesture referent.

Preferably, in the described multi-channel human-computer interaction method based on voice and gesture, in the third step,

Comparing the constraint information of the voice reference object and the gesture reference object constraint information with the characteristic information of the model object in the virtual environment, and determining the reference object of human-computer interaction, the determination of the reference object is realized on the grammar layer,

The command information for the referent is extracted from the constraint information of the above speech referent object in the following ways:

The grammar layer extracts command information from the phonetic reference object constraint information,

Applying command information to referents is accomplished in the following ways:

The semantic layer applies the command information extracted by the syntax layer to the referring object.

Preferably, in the multi-channel human-computer interaction method based on voice and gesture, the multi-channel layered integration model further includes a task slot, and the task slot includes a command entry and a referent entry,

Wherein, the semantic layer applies the command information extracted by the syntax layer to the referent object in the following manner:

The semantic layer fills the command information extracted by the syntax layer into the command table item, and fills the referent object into the referent object table item, the task slot is completely filled, and the multi-channel layered integrated model production system can execute the command.

Preferably, in the multi-channel human-computer interaction method based on voice and gesture, when the task slot is not filled completely, a waiting time is set, and the task slot is filled completely within the waiting time, then continue In this human-computer interaction, if the task slot is not completely filled within the waiting time, the human-computer interaction is abandoned.

Preferably, in the multi-channel human-computer interaction method based on speech and gestures, the command entry includes an action entry and a parameter entry, and the command information for the referent is extracted from the speech reference object constraint information When , the command information includes action information and parameter information.

Preferably, in the multi-channel human-computer interaction method based on voice and gesture, in the first step, when the voice channel receives the first sentence, a human-computer interaction process is started.

Preferably, in the multi-channel human-computer interaction method based on voice and gesture, in the first step, when the voice channel receives a sentence, set the timeout time to receive the input of gesture information of the gesture channel, such as gesture If the input of information exceeds the set timeout period, the human-computer interaction process will be abandoned.

The voice and gesture-based multi-channel human-computer interaction method of the present invention has the following beneficial effects:

(1) Solve the problem of pointing ambiguity in multi-channel human-computer interaction methods based on speech and gestures. When performing three-dimensional interaction in a virtual environment, gestures (from the beginning of pointing recognition to the end of pointing) not only express spatial information, but also carry temporal information. The longer the object stays in the pointing area, the more likely it is to be selected. Therefore, when analyzing the constraint information of gesture referent objects, not only distance statistics but also time statistics should be obtained, so as to reduce pointing ambiguity in 3D interaction. Moreover, in the process of determining the referent object, the model objects in the virtual environment are divided into four categories, and the referent object is compared with a certain type of model object. This method also helps to narrow the search scope of the referent object , reducing the effect of pointing ambiguity.

(2) Solve the problem of unspecified information inference in the multi-channel human-computer interaction method based on speech and gesture. The model objects in the virtual environment are divided into four categories. Among them, the focus object is the referring object determined during the last human-computer interaction, that is, if there is The indicative pronoun "it" can be considered as the referring object of this human-computer interaction is the focused object, thus solving the problem of unspecified information inference.

(3) Provide a multi-channel human-computer interaction method based on voice and gesture. By building a multi-channel hierarchical integration model, four layers are established in the multi-channel hierarchical integration model: physical layer, lexical layer, grammatical layer and semantic layer, and finally the command information and referent objects required for human-computer interaction are filled into tasks The goal of the above-mentioned integration process and the criteria for the success of the integration are based on the integrity of the human-computer interaction task structure. reliability of human-computer interaction.

Description of drawings

FIG. 1 is a schematic diagram of the human-computer interaction process of the multi-channel human-computer interaction method based on voice and gesture according to the present invention.

FIG. 2 is an overall architecture diagram of the reference resolution of the multi-channel human-computer interaction method based on voice and gesture according to the present invention.

FIG. 3 is an overall flow chart of the multi-channel human-computer interaction method based on voice and gesture according to the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings, so that those skilled in the art can implement it with reference to the description.

As shown in Figure 1, Figure 2 and Figure 3, the present invention provides a multi-channel human-computer interaction method based on voice and gesture, comprising the following steps:

Step 1. Build a voice channel and a gesture channel, and input voice information and gesture information to the referent of the human-computer interaction through the voice channel and the gesture channel respectively;

Step 2. Extract the speech reference object constraint information from the above voice information, and extract the gesture reference object constraint information from the above gesture information, wherein the gesture reference object constraint information includes the arrival of any point in the pointing area defined by the current pointing gesture The distance statistics of the pointing center of the pointing gesture and the time statistics maintained by the above-mentioned pointing gesture;

Step 3: Compare the above-mentioned speech referent object constraint information and gesture referent object constraint information with the feature information of the model object in the virtual environment, determine the referent object of human-computer interaction, and extract the referent object from the above-mentioned speech referent object constraint information Command information, which acts on the referent object to complete a human-computer interaction.

As shown in FIG. 1 , the above multi-channel human-computer interaction method based on voice and gesture firstly supports two interaction channels of voice and gesture. The voice recognition module adopts the Microsoft voice recognition engine to map the user's voice commands into text information with time stamps, and the voice analysis module extracts the voice referent object constraint information from it. The gesture channel uses data gloves to obtain joint and position information for gesture recognition. The gesture analysis module accepts pointing gestures and generates pointing object vectors. The multi-channel integration module integrates the information from the voice and gesture channels, realizes the affiliation of the reference during the integration process, and finally generates system executable commands or corresponding prompts.

The present invention adopts a multi-channel layered integration model to realize multi-channel integration. The integration process is task-guided. The goal of integration and the criteria for the success of integration are based on the integrity of the interactive task structure. The ultimate goal is to generate a task structure that can be submitted to the system for execution, including task actions and task functions. Objects and corresponding parameters and other information. Therefore, task slots are defined in the present invention, and task slots are part of the multi-channel hierarchical integration model. The structure of the task slot is divided into three parts, which are the action table item, referent object table item and parameter table item, which can also be called action slot, referent object slot and parameter slot. In fact, both action table items and parameter table items belong to command table items. There can be more than one referent in the referent slot. Currently, the parameter slot can only be filled with position information. Different commands correspond to task slots with different structures. For example, the task slot of the select command has only two entries: action and referent. The process of integration becomes the process of filling task slots. Once the task slots are filled, a complete task that can be executed by the system is formed.

For example, if only the voice input "rotate it" is made without a pointing gesture, the referent cannot be determined. Then when the task slot is filled, the action slot will be filled with "rotation", while the referent slot is empty. At this time, since the waiting time is set, if the task slot is completely filled within the waiting time, that is, a pointing gesture is made within the waiting time, thereby determining the referent, the human-computer interaction will continue. The multi-channel layered integration model will generate system executable commands, and if the task slot is not fully filled within the waiting time, the human-computer interaction will be abandoned.

The multi-channel layered integration model defined in the present invention, as the name suggests, is based on the idea of layering, abstracting channel information from specific device information to semantics to be filled into task slots into physical layer, lexical layer, syntax layer and semantic layer Wait for four floors. The physical layer information is the original information input from the interactive device, and its form is diverse and directly related to the specific interactive device. For example, character string information is input from voice, while sensor information is input from data gloves. The lexical layer is the key layer. It unifies the original information from the device layer, and unifies the input with the same meaning but different forms into the same information representation, thus providing device-independent information to the grammar layer. In the lexical layer, the voice information of the voice channel is abstracted by the voice recognition module and the voice analysis module to generate voice reference object constraint information; at the same time, the gesture information of the gesture channel is abstracted by the gesture recognition module and the gesture analysis module to generate gesture references Object constraint information. The grammatical layer mainly decomposes the information from the lexical layer according to the grammatical norms of human-computer interaction, and decomposes it into forms that conform to each table item of the task slot, so as to prepare for the subsequent semantic fusion. Referential resolution is mainly carried out at the grammatical level. Moreover, the syntax layer also extracts command information from the phonetic referent constraint information. In the semantic layer, the task guidance mechanism is used to fill and complete the task slot. Although the task is related to the specific application, the filling and completion of the task slot is independent of the application.

In fact, the human-computer interaction process can be divided into two strategies, "immediate" and "slow". Acute sub-integration processing begins as soon as the multi-channel input supports some level of integration, and this process can be viewed as event-driven. And the integration of the slow child will not start to process until it has all the inputs or relatively complete inputs. For example, when conducting human-computer interaction, the strategy of the impatient is that the voice input "rotate it", and the multi-channel hierarchical integration model starts to work and begins to process information. The strategy of the slow child is that the voice input "rotate it", and at the same time the pointing gesture is made to point to an object, so that the model can determine the referring object, and the model starts at this time. That is to say, the slow child is providing all the information of a human-computer interaction at one time. Since the user's voice input often appears discontinuous, there is a large time interval in the middle of a complete command of a moving object. Simultaneously limited by the speech recognition engine, the present invention uses the "impatient" strategy, adopts speech drive, and starts a human-computer interaction process when the speech channel receives the first sentence.

The process of identifying the referent is also the process of referential resolution. In the present invention, the reference resolution is based on both speech reference object constraint information and gesture reference object constraint information. The present invention is based on the following two assumptions: (1) the semantics in the speech input is clear, and the present invention mainly focuses on solving the ambiguity of pointing in the multi-channel reference resolution, so it is assumed that the semantics in the speech input is clear and there is no " upper left corner", "middle", "before" and other vague words; (2) "self-centered" references, which can be divided into three types: self-centered, reference-centered, and others-centered . All references in the present invention are self-centered, and there is no such situation as "choose the object on his left" that is centered on other viewpoints.

The present invention adopts a voice-driven integration strategy, and after a sentence is recognized, a multi-channel integration process is triggered. In the multi-channel hierarchical integration model, firstly, the phonetic reference object constraint information is filled into the phonetic constraint set. All model objects in the virtual environment can be assigned identities according to the constraint information of gesture referent objects, and all model objects can be divided into four categories: pointing objects, focusing objects, active objects and silent objects. The pointing object is an object located in the pointing area defined by the current pointing gesture, the focusing object is the referring object determined during the last human-computer interaction, and the activation object is an object located in the visible range except A model object other than the pointing object and the activation object, the silent object is a model object other than the pointing object and the activation object located in an invisible range. Each type of model object corresponds to an initialization matching matrix, which are pointing matrix, focusing matrix, activation matrix and silence matrix.

The present invention adopts the method of perception shape in the process of denotation resolution, and the perception shape is a geometry controlled by the user and can provide information about interactive objects. When the system recognizes that the current gesture is a pointing gesture, it generates a cone attached to the tip of the index finger of the virtual hand (that is, the pointing area defined by the pointing gesture), records the interaction process between the model object and the cone through collision detection, and generates various Statistics data. The weighted average of the statistics is then used to generate pointing priorities. After a pointing interaction is completed, a 2-tuple vector corresponding to the pointing gesture is obtained. The first element of the 2-tuple is the pointing object vector, and the second element is the pointing priority.

The present invention defines two statistics of time series T _rank and distance series D _rank . The longer the time in the perceived shape and the closer the distance to the pointing center (the virtual hand index fingertip), the higher the priority of the model object.

The calculation process of T _rank is shown in the following formula, where T _object represents the time of a certain model object in the cone, and T _period represents the existence time of the cone during a certain interaction process (that is, the duration of the pointing gesture).

$T_{\mathrm{rank}}=\frac{T_{\mathrm{object}}}{T_{\mathrm{period}}},$ 0<T _rank ≤1

The calculation process of D _rank is shown in the following formula, where D _object represents the distance from the center of a model object to the pointing center, and D _max is the farthest distance from the model object in the cone to the pointing center.

${\mathrm{D.}}_{\mathrm{rank}}=1-\frac{{\mathrm{D.}}_{\mathrm{object}}}{{\mathrm{D.}}_{\max }},$ 0<D _rank ≤1

The pointing priority P _rank is obtained by the weighted average of the above two statistics, and its calculation method is as follows:

P _rank =T _rank *λ+D _rank *(1-λ), 0≤λ≤1

Since interactive devices are not designed to work in a cooperative manner, integration across channels must rely on time correlation. Therefore, after obtaining the pointing priority P _rank through perceptual shape calculation, the current time should be recorded for the later stage of multi-channel integration. Since the task slot has a waiting time setting for further information input, the value of this waiting time should take into account the time required for further input of gesture information and completion of the referral resolution process together with voice information.

After the pointing priority and pointing object vector are obtained above, the pointing matrix, focusing matrix, activation matrix, and silence matrix will be compared and searched one by one, and the model objects in the four matrices have corresponding states. At each stage, when referring to the model objects located in the same matrix, the state of the model objects is quantified by the matching function Match(o, e).

The matching function is constructed as follows:

$\mathrm{<span\; class="notranslate">\; match</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; [</span>\underset{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; Semantic</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; Temp</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span>$

Among them, o represents the model object and e represents the reference. P means the pointing state, F means the focusing state, A means the active state, E means the quiet state, and S means the state of the current object. The following are the various components of Match(o, e):

(1) P(o|S) and P(S|e)

(M为聚焦对象的个数)，

(N为激活对象的个数)，

(L表示虚拟环境中所有模型对象的个数)。P(S|e)是当指称为e时指称对象状态为S的概率。P(o|S) represents the probability that an object o is selected given a cognitive state S, and is used to measure the impact of gesture channels on referential resolution. The specific calculation method is: P(o|P)=P _rank ;

(M is the number of focus objects),

(N is the number of activated objects),

(L represents the number of all model objects in the virtual environment). P(S|e) is the probability that the referent state is S when the referent is e.

(2) Semantic (o, e)

Semantic(o, e) represents the semantic compatibility between the model object o and the reference e, which is used to measure the influence of the speech channel on the reference resolution, and its construction is as follows:

$\mathrm{<span\; class="notranslate">\; Semantic</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\frac{{\mathrm{<span\; class="notranslate">\; Attr</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; K</span>}}$

The present invention classifies both identifiers and semantic types into the attribute Attr _k , and Attr _k (o, e) is 0 when both o and e have attribute k and the two values are not equal, and is 1 in other cases. K is the total number of attributes referring to the object.

(3)Temp(o,e)

Temp(o, e) represents the temporal compatibility between the model object o and the reference e, and is used to measure the impact of time on reference resolution. It is a piecewise function:

When o and e are in the same interaction, the calculation process of Temp(o, e) is as follows:

Temp(o,e)=exp(-|Time(o)-Time(e)|)

When o and e are in different interactions, the calculation process of Temp(o, e) is as follows:

Temp(o,e)=exp(-|OrderIndex(o)-OrderIndex(e)|)

Among them, Time(o) is the time when the pointing gesture occurs, Time(e) is the time when the reference occurs, and the unit is second; OrderIndex(o) indicates the order of o in the sequence of pointing gestures, and OrderIndex(e) indicates the order of e in the sequence of references priority. Temp(o,e)=1 for objects in focus, activation or silence.

When the reference and the model object in a certain state (that is, in a certain matrix) are compared and matched, the reference object is confirmed.

In the multi-channel human-computer interaction method based on voice and gestures, the model objects in the virtual environment are divided into four categories: pointing objects, focusing objects, activation objects and silent objects, and the pointing objects are located in the current pointing gesture Objects within the defined pointing area, the focus object is the reference object determined during the last human-computer interaction, and the activation object is a model object located within the visible range other than the pointing object and the activation object , the silent object is a model object located in an invisible range except for the pointing object and the activation object. The feature information of the active object, the active object, and the silent object are compared to determine the referent object of the human-computer interaction.

In the multi-channel human-computer interaction method based on voice and gesture, in the second step, extracting the voice reference object constraint information from the above voice information and extracting the gesture reference object constraint information from the above gesture information in the following manner Achieved: build a multi-channel layered integration model, the multi-channel layered integration model includes four layers, respectively physical layer, lexical layer, syntax layer and semantic layer, wherein, the physical layer receives voice channel and The voice information and gesture information input by the gesture channel, the lexical layer includes a voice recognition and analysis module and a gesture recognition and analysis module, the voice recognition and analysis module analyzes the voice information of the physical layer into voice reference object constraint information, and the gesture recognition The parsing module parses the gesture information of the physical layer into constraint information of the gesture referent.

In the multi-channel human-computer interaction method based on voice and gesture, in the third step, the above-mentioned voice reference object constraint information and gesture reference object constraint information are compared with the feature information of the model object in the virtual environment to determine the human The referent object of machine-computer interaction, the determination of the referent object is realized on the grammar layer, and the command information for the referent object is extracted from the above-mentioned phonetic referent object constraint information in the following manner: the grammar layer extracts the command information from the speech Extracting command information from the constraint information of the referent object, and applying the command information to the referent object is realized in the following manner: the semantic layer applies the command information extracted by the syntax layer to the referent object.

In the multi-channel human-computer interaction method based on speech and gestures, the multi-channel layered integration model further includes a task slot, and the task slot includes a command entry and a referent entry, wherein the semantic layer will The command information extracted by the grammatical layer acts on the referring object in the following manner: the semantic layer fills the command information extracted by the grammatical layer into the command table item, fills the referring object into the referring object table item, and the task slot Completely populated, the multi-channel layered integrated model production system can execute commands.

In the multi-channel human-computer interaction method based on voice and gestures, when the task slot is not filled completely, a waiting time is set, and the task slot is filled completely within the waiting time, then the human-machine interaction is continued. If the task slot is not completely filled within the waiting time, the human-computer interaction is abandoned.

In the multi-channel human-computer interaction method based on speech and gestures, the command entry includes an action entry and a parameter entry, and when the command information for the referent is extracted from the speech reference object constraint information, the Command information includes action information and parameter information.

In the multi-channel human-computer interaction method based on voice and gesture, in the first step, when the voice channel receives the first sentence, a human-computer interaction process is started.

In the described multi-channel human-computer interaction method based on voice and gesture, in the step 1, when the voice channel receives a sentence, the timeout period is set to receive the input of the gesture information of the gesture channel, such as the input of the gesture information exceeds If the set timeout time is exceeded, the human-computer interaction process is abandoned.

Although the embodiment of the present invention has been disclosed as above, it is not limited to the use listed in the specification and implementation, it can be applied to various fields suitable for the present invention, and it can be easily understood by those skilled in the art Therefore, the invention is not limited to the specific details and examples shown and described herein without departing from the general concept defined by the claims and their equivalents.

Claims (1)

1. the multimodal human-computer interaction method based on voice and gesture, is characterized in that, comprises the following steps:

Step 1, structure voice channel and gesture passage, and the referent of man-machine interaction is carried out to the input of voice messaging and gesture information by voice channel and gesture passage respectively;

When voice channel receives first statement, start interactive process one time;

When voice channel receives a statement, time-out time is set, as the input of gesture information exceeds set time-out time, abandon this interactive process;

Step 2, extract voice referent constraint information from above-mentioned voice messaging; extract gesture referent constraint information from above-mentioned gesture information; wherein, described gesture referent constraint information comprises that any point in the indication zone that current indication gesture limits arrives the distance statistics amount at the indication center of giving directions gesture and the time statistic that above-mentioned indication gesture maintains;

Extract voice referent constraint information and extract gesture referent constraint information from above-mentioned voice messaging from above-mentioned gesture information and be achieved in the following ways:

Build hyperchannel layering Integrated Models; described hyperchannel layering Integrated Models includes four layers; be respectively Physical layer, morphology layer, grammer layer and semantic layer; wherein; described Physical layer receives voice messaging and the gesture information of being inputted by voice channel and gesture passage respectively; described morphology layer includes speech recognition parsing module and gesture identification parsing module; described speech recognition parsing module resolves to voice referent constraint information by the voice messaging of Physical layer, and described gesture identification parsing module resolves to gesture referent constraint information by the gesture information of Physical layer;

Step 3, model object in virtual environment is divided into the indication object, focal object, activate object and quiet object four classes, described indication object is the object that is positioned at the indication zone that current indication gesture limits, the referent of described focal object for being determined in upper once interactive process, described activation object is the model object except giving directions object and activation object that is positioned at visual range, described quiet object is the model object except giving directions object and activation object that is positioned at not visible scope, by above-mentioned voice referent constraint information and gesture referent constraint information in order one by one with above-mentioned indication object, focal object, activate object, the characteristic information of quiet object is contrasted, determine the referent of man-machine interaction, extract the command information to referent from above-mentioned voice referent constraint information, command information is acted on to referent, complete a man-machine interaction,

The characteristic information of model object in above-mentioned voice referent constraint information and gesture referent constraint information and virtual environment is contrasted, determine the referent of man-machine interaction, the definite of described referent realize on described grammer layer,

The command information of extracting referent from above-mentioned voice referent constraint information is achieved in the following ways:

Described grammer layer extracts command information from voice referent constraint information,

Command information is acted on to referent to be achieved in the following ways:

The command information that described semantic layer extracts the grammer layer acts on referent;

Described hyperchannel layering Integrated Models also includes the task groove, and described task groove comprises order list item and referent list item,

The command information that wherein said semantic layer extracts the grammer layer acts on referent and carries out in the following manner:

The command information that described semantic layer extracts the grammer layer is inserted the order list item, and referent is inserted to the referent list item, and described task groove is filled complete, described hyperchannel layering Integrated Models production system executable command;

In the situation that described task groove is not filled is complete, the stand-by period is set, described task grain is filled complete within the stand-by period, continues this man-machine interaction, and described task groove is not filled complete within the stand-by period, abandons this man-machine interaction;

Described order list item includes action list item and parameter list item, and while extracting the command information to referent in described voice referent constraint information, described command information comprises action message and parameter information.

Images