KR101567995B1 - Method, appratus and computer-readable recording medium for providing user interface which enalbes interaction - Google Patents

Abstract

According to an aspect of the present invention, there is provided a method for recognizing a user's interaction using an electronic device, the method comprising: (a) generating a plurality of recognition results using a plurality of recognition algorithms; ) The method includes generating the assurance value of the interaction type indicating the semantic strength of the interaction type by combining the plurality of recognition results according to the definition of the interaction type. According to the present invention, even if a general user who does not have detailed knowledge on the recognition algorithm of the lower layer or the perception inference through the loosely coupled development method understands the abstract meaning and the semantic intensity of the interaction type, An effect that can be applied to the present invention is achieved.

Description

[0001] METHOD, APPARATUS AND COMPUTER READABLE RECORDING MEDIUM FOR PROVIDING USER INTERFACE WHICH ENALBES INTERACTION [0002]

The present invention relates to a method, an apparatus, and a computer-readable recording medium for providing a user interface for supporting an interaction, and more particularly, to a method and apparatus for generating a plurality of recognition results using a plurality of recognition algorithms, An apparatus, a system, and a computer-readable recording medium that can provide an assured value of an interaction type indicating a semantic strength of an interaction type by combining recognition results according to definition of an interaction type.

There is a growing demand for interaction interfaces between humans and computing environments, and interaction interface technologies such as natural interaction devices and SDK are being developed.

When an application related to an interaction interface is developed, the interaction used in the application is defined and applied through a recognition algorithm. In defining and applying the interaction through the recognition algorithm, recognition algorithms and applications are generally developed in tightly coupled fashion. In other words, in general, conventional human sensing techniques are implemented such that a function corresponding to a detected operation is performed when a predetermined specific operation is detected. This closely linked form of development leads to a reduction in the reusability of core modules such as recognition technology. That is to say, a closely linked type of development method is a method in which, when an operation purpose or an environment related to an application is changed, the code of the recognition algorithm is significantly modified in accordance with the purpose of the changed operation or the changed environment or a series of parameters There is a problem of inefficiency such that they must be adjusted.

For example, recognition algorithms provided in conventional recognition libraries, such as OpenNI and Intel Perceptual Computing SDK, have the problem that they are device dependent or limited in kind. In addition, the conventional recognition library has a problem in terms of expandability and reusability. In other words, the conventional recognition library has a problem that it is difficult to apply a reasonable algorithm other than the recognition algorithm that is simply provided in the recognition library, and it is difficult to accurately reflect the intention of the person.

Accordingly, the present inventors have found that by adding a new abstraction layer on a layer of an existing recognition module including a recognition algorithm and by using perceptual inference on the results of recognition algorithms, And loosely coupled implementation with respect to the application.

It is an object of the present invention to solve all the problems described above.

The present invention also provides a loosely coupled implementation between the recognition module and the interaction type, by adding an abstraction layer of interaction type above the hierarchy of recognition modules and using perceptual reasoning for recognition results of recognition algorithms It is another purpose to support loosely coupled implementations between interaction types and applications.

In order to accomplish the above object, a representative structure of the present invention is as follows.

According to an aspect of the present invention, there is provided a method for providing a user interface for supporting an interaction, comprising: (a) the electronic device generating a plurality of recognition results using a plurality of recognition algorithms; and (b) And generating an assertion value of the interaction type indicating the semantic intensity of the interaction type by combining the plurality of recognition results according to the definition of the interaction type.

According to another aspect of the present invention, there is provided an electronic apparatus for providing a user interface for supporting an interaction, comprising: a plurality of recognition units for generating a plurality of recognition results using a plurality of recognition algorithms; To generate an assurance value of the interaction type indicating the semantic strength of the interaction type.

In addition, there is further provided another method, apparatus, system for implementing the invention and a computer readable recording medium for recording a computer program for executing the method.

According to the present invention, even if a general user who does not have a specific knowledge on a recognition algorithm of a lower layer or a perceptual reasoning or the like understands an abstract meaning and an intensity of an interaction type through a loosely coupled development method, The effect that can be applied to an interaction application is achieved.

Further, according to the present invention, a developer who studies a recognition algorithm or perceptual reasoning through a loosely coupled development method freely adds or combines a recognition algorithm and a reasoning algorithm developed by the user in an interaction application, The effect of changing the configuration or defining a new interaction type is achieved.

In addition, according to the present invention, the abstracted interaction type is not limited to a specific application, but can be used redundantly for a plurality of applications requiring similar interactions, thereby providing an effect of providing an interaction type having high reusability.

In addition, according to the present invention, various different recognition algorithms and reasoning algorithms can be added or changed for an interaction type having the same meaning, and thus an effect of providing a highly-scalable interaction type is achieved.

1 illustrates an interaction according to an embodiment of the present invention.
Figure 2 shows a classification of an interaction type according to an embodiment of the present invention.
3 is a diagram showing an internal configuration of an electronic device according to the present invention.
FIG. 4 illustrates a user interface engine according to an embodiment of the present invention.
FIG. 5 shows a recognition unit used for an interaction type and an interaction type used in an application according to an embodiment of the present invention.
6A illustrates an example of an asserted value of an interaction type according to an embodiment of the present invention.
6B illustrates an example of an asserted value of an interaction type according to an embodiment of the present invention.
Figure 7 illustrates adaptive perceptual reasoning in accordance with an embodiment of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily carry out the present invention.

Interaction  And Interaction  type

1 illustrates an interaction according to an embodiment of the present invention.

According to one embodiment of the present invention, an interaction may mean an action that may affect one another between two or more objects or two or more people. Interaction can be classified into human-world interaction (or human-object interaction) and human-human human-human interaction in Human vs. World. In Fig. 1, as an example of a human-world interaction, an interaction related to another person (i.e., the other person) and an object (i.e., a chess horse) is shown.

According to an embodiment of the present invention, an interaction type may mean a high-level abstract description capable of describing an interaction.

In general, with respect to interactions, existing applications have limited access to the level of awareness. On the other hand, according to one embodiment of the present invention, interpretation is required for meaningful interaction. For example, as shown in FIG. 1, human-world interactions may include not only levels of awareness, but also other levels related to interpretation, intention, and behavior.

Nonverbal messages can be used to: 1) emphasize or modify language expressions, 2) convey information about emotional states, 3) define or enhance relationships among people, and 4) . For example, in emphasizing language expressions, one can enthusiastically nod his head saying "yes" to emphasize sympathy for others. Also, in the modification of language expression, a person shrugs his shoulders and says "I am really okay," and by making a sad expression, he can imply a different state from the meaning of his words. Also, in the flow of dialogue, in a coordination, a person can indicate to others via the signal that he has done something to say or that there is something left to do. In other words, in understanding the messages exchanged between human beings and human beings, or between humans and objects, nonverbal cues (Behavioral signal) are generated as much as the weight of the existing speech language, Can play a big role.

According to one embodiment of the present invention, interpretation of human-human interaction or human-object interaction may be considered as interpretation of the behavioral signal. The classification of the interaction type according to the embodiment of the present invention can be defined according to the classification scheme of the behavior signal. Below, the classification of the interaction type is illustrated with reference to Fig.

Figure 2 shows a classification of an interaction type according to an embodiment of the present invention.

As described above, the interaction type can be classified into human-to-human interaction and human-to-human interaction. For example, the category of human-to-body interaction may include a descriptor (Illustrator) and a manipulator. In addition, the category of human-to-human interaction may include a descriptor, a manipulator, a Regulator, and an Affective State. In Figure 2, the categories of human-to-human interaction are shown in more detail. For example, the category of the descriptor may include Pointing, Ideograph, and EyeGaze. In addition, the category of the navigator may include Approach, Recede, Stare, Grasp, Translation, and Rotation. The example of the categories of human-to-human interaction described with reference to Fig. 2 can be extended to the category of human-to-human interaction.

Below, the categories of interaction are described in detail.

(i) The descriptor may be an act describing verbal messages. Examples of descriptors include pointing or drawing, and Hand Movement with Accompany accompanied by a phrase of Over and over again. And there is a nod in a specific direction when you say "Over there".

(ii) The navigator may be an action used to take action on objects in the environment. Examples of navigators are Select, Gather, Move, and Rotate. In addition, the maneuver may be manipulative, especially called adaptors. Examples of self-guided behaviors include Scratching, Adjusting uncomfortable glasses, Tapping pens, Twirling Hair, and Playing with Jewelry. There is a representative a psychological need such as biting nails.

(iii) The Adjuster may be an act that allows us to maintain, ask or deny our conversation with others. Examples of adjusters include, but are not limited to, Exchange of a look, head nods, head shake, uh-huh, and mm-mm. There are short sounds and expressions of interest or boredom. The Adjuster may allow the other person to adjust to his or her utterance to reflect the level of interest or consent. Many people will find it difficult to maintain a conversation without receiving feedback.

(iv) An emotional state can be an act that describes an emotional state.

Below, an example of an interaction type, the interactions in a chess game are illustrated.

(i) As a manipulator, (look at) (look straight at), (look at chess), (select) (chess), (movement) and (movement) '.

There are also Self-Manipulators (so-called adapters), such as a mask (a hand-restrained chin rest) and a nail bit.

(ii) As Adjuster, 'Look straight at', 'Shake up' and 'Shake' and 'Shake'. .

(iii) descriptors, such as pointing, nodding up and down the head. As an example of a nod, there is a nod to the head in a specific direction, saying "over there."

(iv) Emotional states, facial expression. Examples of facial expressions are Surprise, Happy, Sad, Angry, Exciting, and Boring.

Configuration of the device

3 is a diagram showing an internal configuration of an electronic device according to the present invention.

3, the electronic device 300 may include an interaction type unit 310, a reasoning unit 320, and a plurality of recognizing units 330. As shown in FIG. Here, the electronic device 300 may further include a sensor 340.

According to an embodiment of the present invention, the interaction type unit 310, the reasoning unit 320, and the plurality of recognizing units 330 may be program modules in which at least a part of them communicate with an external system. Such program modules may be included in the electronic device 300 in the form of an operating system, application program modules, and other program modules, and may be physically stored on a variety of known memory devices. Also, at least some of these program modules may be stored in a remote storage device capable of communicating with the electronic device 300. Such program modules include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform particular tasks or perform particular abstract data types as described below in accordance with the present invention.

According to an embodiment of the present invention, the electronic device 300 can recognize a user's interaction using an interaction type unit 310, a reasoning unit 320, a plurality of recognition units 330, and a sensor 340 have.

(i) First, according to an embodiment of the present invention, the sensor 340 may be an image capturing device for capturing an image. For example, the sensor 340 may be a color camera, a depth camera, or an infrared camera. The sensor 340 may be a sound recording device for recording sound. For example, the sensor 340 may be a microphone. The sensors 340 may be one or more, and one or more sensors may capture images or record sounds, respectively. The photographed image or recorded sound can be used as input data by recognition algorithms to be described later.

(ii) Next, according to an embodiment of the present invention, the interaction type unit 310 may define an interaction type. In other words, the interaction type unit 310 can manage the data required for the definition of the interaction type.

According to an embodiment of the present invention, the interaction type unit 310 may define an interaction type having a meaning such as 'starvation', 'selection', 'movement', and 'rotation'. The interaction type unit 310 may define an interaction type more specifically expressed in addition to the interaction type expressed in basic units such as 'staring', 'selection', 'moving' and 'rotation'. That is to say, the interaction type unit 310 can define one or more lower interaction types for the defined interaction type. For example, the interaction type unit 310 is a concrete interaction type for the interaction type having the meaning of 'stares', and can be classified into 'interaction type', 'looking at a person', 'looking at an object' You can define interaction types such as 'staring straight' and 'staring staring'.

According to an embodiment of the present invention, an interaction type can be defined and classified as a minimum meaning unit of an interaction. The abstraction interaction type may be a combination of one or more minimum semantics, and the defined interaction type may be applied in an application operating in the electronic device 300 or the like.

According to one embodiment of the present invention, the definition of the interaction type may be in the form of applying adaptive perceptual reasoning to the results of a plurality of recognition algorithms.

(iii) Next, according to an embodiment of the present invention, the plurality of recognizing units 330 may generate a plurality of recognition results using a plurality of recognition algorithms. Each recognition unit of the plurality of recognition units 330 can generate a recognition result using a recognition algorithm. The plurality of recognizing units 330 may use different recognition algorithms.

According to one embodiment of the present invention, each recognition algorithm may use at least one of the image captured by the sensor 340 and the recorded sound as input data to generate a recognition result for the input data. When there are a plurality of sensors 340, each recognition unit of the plurality of recognition units 330 can receive input data from a part of sensors outputting input data of a form required by a recognition algorithm among a plurality of sensors.

According to an embodiment of the present invention, the plurality of recognizing units 330 may add an existing recognition algorithm or a recognition algorithm developed by a user of the electronic device 300 to a plurality of recognition algorithms.

(vi) Next, according to one embodiment of the present invention, the reasoning unit 320 may generate a confidence value of the interaction type by combining a plurality of recognition results according to the definition of an interaction type. In other words, the plurality of recognizing units 330 may be subjects performing recognition algorithms that provide the recognition results required to generate the confidence value of the interaction type according to the definition of the interaction type.

According to an embodiment of the present invention, the confidence value of the interaction type may indicate the meaning strength of the interaction type. The semantic intensity of the interaction type may be an intensity representing at least one of the image and sound used in the electronic device 300 indicating the interaction that the interaction type means and the probability that the image or sound will represent the interaction represented by the interaction type Lt; / RTI > In other words, the confidence value of the interaction type may be a value indicating the degree of confidence in the interaction. Here, the image may be an image photographed by the sensor 340, or may be an image used in an application or an image output from an application. The sound may be the image recorded by the sensor 340, and the sound used or output by the application. Using the confidence values of interaction type and interaction type, the developer can easily apply the interaction type to the application using only the abstract level meaning of the interaction type without specific understanding of the content or result of the recognition algorithm.

According to one embodiment of the present invention, the confidence value may be a value between 0 and 1, for example, and a value between 0 and 1 may indicate confidence. The higher the confidence value, the more likely it is that the image or sound is more likely to have content consistent with the meaning of the interaction type.

According to an embodiment of the present invention, the reasoning unit 320 may generate an assertion value of an interaction type by quantitatively combining a plurality of recognition results generated by a plurality of recognition algorithms. In addition, the reasoning unit 320 may generate an assertion value of an interaction type by applying an inference reasoning adaptive to a plurality of recognition results. The inference unit 320 may generate an assertion value of the interaction type by applying an adaptive reasoning combination to a plurality of recognition results based on at least one of probabilistic analysis, structural analysis, and temporal analysis. That is to say, the interaction type is constituted by a quantitative combination of a plurality of recognition results generated by a plurality of recognition algorithms, an adaptive reasoning combination based on probabilistic analysis, structural analysis, and temporal analysis on a plurality of recognition results .

According to an embodiment of the present invention, the reasoning unit 320 combines predetermined recognition results selected according to the definition of the interaction type among the plurality of recognition results generated by the plurality of recognition algorithms, thereby obtaining the confidence value of the interaction type Can be generated. In other words, the reasoning unit 320 can generate the confidence value of the interaction type using some recognition results related to the interaction type among the entire recognition results generated by the plurality of recognition algorithms. For the interaction type, the interaction type unit 310 can define recognition results related to the interaction type, and can define a method for combining the recognition results.

According to one embodiment of the present invention, the image used to generate the confidence value may be a series of images over time. That is, the image may include one or more frames. The inference unit 320 may generate an assertion value of an interaction type for each frame of the image or for each frame period. The inference unit 320 may generate confidence values of one or more frames of the image, an interaction type for each frame or for every frame period. The confidence value of the interaction type can change over time.

According to one embodiment of the present invention, the interaction type may be plural. The interaction type unit 310 may define a plurality of interaction types, and the reasoning unit 320 may generate a confidence value for each of a plurality of interaction types. The reasoning unit 320 combines the recognition results selected for each interaction type of a plurality of interaction types according to the definition of each interaction type among a plurality of recognition results available for the electronic device 300, Can be generated. The inference unit 320 may generate confidence values of each interaction type for each of the plurality of interaction types, for each frame or for each arbitrary frame period.

According to one embodiment of the present invention, the interaction type unit 310 can change the definition of the interaction type, and the reasoning unit 320 can change the combination of the plurality of recognition results according to the change of the definition of the interaction type , A plurality of recognition algorithms used for generation of the assurance value of the interaction type, and a plurality of recognizers corresponding to the plurality of recognition algorithms. That is to say, combinations of recognizers (or combinations of recognition algorithms) used to generate confidence values of the interaction type over time can be changed. Through changing the definition of the interaction type and changing the combination of the plurality of recognition results, the combination of the plurality of recognition results may not be limited to a specific combination. In addition, the interaction type unit 310 and the reasoning unit 320 can expand the interaction type by changing the definition of the interaction type and changing the combination of the plurality of recognition results. For example, when a new recognition algorithm is added to a plurality of existing recognition algorithms, the interaction type unit 310 and the reasoning unit 320 may define a combination of a plurality of recognition results and a plurality of recognition results Can be changed. In addition, the number of cases is not limited in the definition of the interaction type and the combination of the plurality of recognition results.

User interface engine

FIG. 4 illustrates a user interface engine according to an embodiment of the present invention.

In accordance with one embodiment of the present invention, the user interface engine 400 may be a Typed interaction interface engine that supports typing of the interaction. Here, the user interface engine 400 may be a framework related to the user interface. The electronic device 300 may provide the user interface engine 400 to an application running inside the electronic device 300 or to an application running outside the electronic device 300.

2, the user interface engine 400 may include an interaction type layer 410, a perceptual reasoning layer 420, and a recognition algorithm layer 430. The interaction type layer 410 may be an upper layer of the perception inference layer 420 and the perception inference layer 420 may be an upper layer of the recognition algorithm layer 430.

The interaction type layer 410, the perceptual reasoning layer 420 and the recognition algorithm layer 430 correspond to the interaction type unit 310, the reasoning unit 320, and the interaction type unit 310 described above with reference to FIG. 3, And can correspond to the plurality of recognizing units 330, respectively. In other words, the interaction type unit 310, the reasoning unit 320, and the plurality of recognition units 330 may implement the interaction type layer 410, the perception inference layer 420, and the recognition algorithm layer 430, respectively.

According to an embodiment of the present invention, the user interface engine 400 composed of three layers can be used to improve existing development methods related to the interaction. The three layers may have a loosely coupled structure. Here, the loosely coupled structure of the three layers may be conceptually similar to the hierarchical structure in a programming language or library. For example, in the library, functions dependent on the operating system in the lower layer may be implemented, and functions of the operating system abstraction layer that is independent of the operating system in the upper layer may be implemented. In addition, functions of the Operating System Abstraction Class can be implemented using functions of the operating system abstraction layer. An application can be implemented using functions of the operating system abstraction class, which are provided regardless of the operating system-dependent lower layer. In a concept similar to the hierarchical layers of the library as described above, an interaction type independent of the recognition algorithm and perceptual reasoning corresponding to the lower layer can be provided.

According to an embodiment of the present invention, a user can easily develop an application using provided interaction types without a detailed understanding of the lower layer of the recognition algorithm layer 430 and the perception inference layer 420.

According to an embodiment of the present invention, a common interface module may be provided in the recognition algorithm layer 430. [ The common interface module can provide a method for unifying the type of input to the recognition algorithm and the type of output from the recognition algorithm according to the proposed guidelines. In other words, the plurality of recognition algorithms can be used by the reasoning unit 320 through the public interface module, and the plurality of recognition results generated by the plurality of recognition algorithms can be generated through the common interface module to generate confidence values of the interaction type And may be provided to the reason speculation unit 320. Through the common interface module, various types of recognition results generated by different recognition algorithms can be provided to the reasoning unit 320 in a consistent form. Further, through the common interface module, various types of input data generated by different sensors can be provided to the recognition algorithm. In addition, various existing recognition algorithms developed in various environments through the common interface module can be easily added to the electronic device 300.

According to one embodiment of the invention, the common interface module may be an interface to input data, output data and parameters of the recognition algorithm. For example, the input data may include color images, depth images, sounds, and the like as data required to perform the recognition algorithm. The output data is data required to store the result of the performance of the recognition algorithm, and may include face area coordinates and the like. The parameter is a value used for adjusting the performance of the recognition algorithm, and may include a threshold value for edge determination, and the like.

For example, a recognition algorithm may inherit an abstract class having a common function of a common interface, and override the inherited abstract class so that the user interface engine is automatically executed. As the user interface engine is executed, the recognition result of the recognition algorithm can be provided to the reasoning unit 320. [

Interaction  Examples of Type Definitions

FIG. 5 shows a recognition unit used for an interaction type and an interaction type used in an application according to an embodiment of the present invention.

At the top of FIG. 5 there are shown examples of applications: (i) Put that there, (ii) Virtual shopping, (iii) Virtual chess, and (iv) . In addition, the interaction type used in each application is shown. In Figure 5, the interaction type is outlined as 'IT' (Interaction Type). For example, 'IT 1' can represent the first type of interaction. For example, in a virtual shopping application, IT 1, IT 6, IT 7, and IT 8 may be used.

5, the recognition units used for generating the meaning of each interaction type and the assurance value of the interaction type are shown. In FIG. 5, the recognition unit is outlined as a 'PU' (Perception Unit). For example, 'PU 1' may represent the first recognition unit.

As shown in FIG. 5, for example, the meaning of IT 1 may be 'look at' and PU 7, PU 4 and PU 5 may be used to generate a confidence value of IT 1. That is to say, the recognition results of recognition algorithms of PU 7, PU 4 and PU 5 can be used to generate confidence values of IT 1.

Referring again to the bottom of FIG. 5, an example of recognition units is shown. For example, the plurality of recognition units of PU 1 to PU 7 each include a body tracking unit for performing a recognition algorithm for tracking a body, a hand tracking unit for performing a recognition algorithm for tracking a hand, A head position estimating unit for performing a recognition algorithm for estimating a head position, a gaze tracking unit for performing a gaze tracking algorithm, a tracking unit for tracking an object, A face recognition unit for performing a recognition algorithm, and a face recognition unit for performing a face recognition recognition algorithm.

For example, face recognition, head posture estimation, and gaze tracking recognition algorithms can be used to generate confidence values of the 'look at' interaction type, and according to the definition of the interaction type of 'look at' Recognition results of selected recognizers (or selected recognition algorithms) of face recognition, head position estimation and gaze tracking among the parts (or recognition algorithms) can be used to generate confidence values of the 'look' have.

Example of Output of Confidence Value

6A illustrates an example of an asserted value of an interaction type according to an embodiment of the present invention.

In FIG. 6A, the names and confidence values of a plurality of interaction types for the illustrated image are shown.

Also, in Fig. 6A, a plurality of recognition algorithms for generating an asserted value of an interaction type are shown to include a recognition algorithm of a hand tracking and a recognition algorithm of an object tracking for a chess manipulation. In addition, there are shown a plurality of interaction types belonging to the category of manipulation, the interaction types of approach, gaze (i.e., gaze), pick, movement and rotation, and assurance values of each interaction type.

As shown in FIG. 6A, the likelihood that the depicted image will represent approach, stare, pick, and move is fairly high, and the likelihood of relatively rotating is low.

6B illustrates an example of an asserted value of an interaction type according to an embodiment of the present invention.

In Figure 6B, the names and confidence values of a plurality of interaction types for the depicted image are shown.

Also, in FIG. 6B, a plurality of recognition algorithms for generating an assertion value of an interaction type are shown to include a face recognition algorithm for a face. Further, there are shown a plurality of interaction types belonging to the category of the manipulation, the interaction types of the approach, the gaze, the pick, the movement and the rotation, and the assurance values of the respective interaction types.

As shown in FIG. 6B, there is little likelihood that the illustrated image has a relatively high likelihood of approaching and gazing, and that it exhibits relative movement, movement, and rotation.

When the image of FIG. 6A and the image of FIG. 6B are images at a specific time point among a series of images, as shown in FIGS. 6A and 6B, for one image, Confidence values can be generated. In addition, the confidence value of the interaction type may change over time, and the recognition algorithm (or recognition section) used to generate the confidence value of the interaction type may also change over time.

Adaptive  Perception reasoning

Figure 7 illustrates adaptive perceptual reasoning in accordance with an embodiment of the present invention.

According to one embodiment of the present invention, as described above, when a plurality of recognition results of a plurality of recognition algorithms are combined and used to calculate a confidence value of one interaction type, a plurality of recognition results are generated using a common interface module In addition to being unified, more complex interpretations may be required for multiple recognition results.

According to an embodiment of the present invention, the interaction type unit 310 may perform probabilistic, structural, and temporal inference on the plurality of recognition results in a quantitative combination of a plurality of recognition results generated by a plurality of recognition algorithms By adding interpretations, we can construct interaction types with high dimensional abstract meanings. The inference unit 320 performs probabilistic, structural, and temporal inference analysis on the plurality of recognition results in a quantitative combination of a plurality of recognition results generated by a plurality of recognition algorithms according to the definition of the interaction type In addition, the confidence value of the interaction type can be calculated. For example, the recognition results of the recognition algorithms constituting the interaction type having the meaning of 'staring' are not directly quantitatively combined, but are used for probabilistic, structural and temporal reasoning analysis of the reasoning unit 320 , And the result of the inference analysis can be expressed by the confidence value of the interaction type.

According to an embodiment of the present invention, the quantitative result may be data output from the recognition unit (or recognition algorithm) and input to the reasoning unit 320. [ For example, the recognition algorithm related to face recognition can represent the position coordinates of the basic elements of the face such as the face center, eye, and nose by quantitative values through face recognition. In addition, the recognition algorithm related to the head posture estimation can quantitatively represent the center coordinates of the head and the inclined angles of the x-axis, the y-axis and the z-axis with respect to the center coordinates, respectively. The quantitative results may include these quantitative values generated by the recognition algorithm.

According to one embodiment of the present invention, temporal analysis, probabilistic analysis, and hierarchical analysis can be used organically related to each other within adaptive perceptual reasoning such as Bayesian based reasoning. In the following, temporal reasoning, probabilistic reasoning, and hierarchical reasoning according to an embodiment of the present invention are described in turn.

Generally, in the human behavior recognition process, human behavior is classified using accumulated modeling data while tracking the posture of a human appearing in sequential images. For example, in the Bayesian model or the Hidden Markov Model, each of human behaviors can be represented as a state, and human behavior can be distinguished according to the transition between states. At this time, the states are connected in the context of the same time as the previous state and the subsequent state. According to one embodiment of the present invention, the temporal analysis may include inferences that comprehensively consider the connected states in terms of time.

According to an embodiment of the present invention, the states mentioned in the temporal analysis described above may be connected with a certain probability. In other words, probabilistic analysis may involve calculating and learning the probability of a transition between each state (say, small behaviors) that make up the behavior in recognizing a behavior with a single meaning. For example, probabilistic analysis may include inference analysis that predicts where a feature obtained from one image (or frame) will appear in the next image (or frame).

According to one embodiment of the present invention, the hierarchical analysis may include a hierarchical reasoning analysis inferring a series of behaviors hierarchically represented in human behavior. For example, in order to catch a chess piece of a chessboard, a person is finally 'picked' as he tilts forward, moves his shoulders, moves his arms, moves his hands, and moves his fingers. These series of movements may not be independent of each other, but may have hierarchical steps or steps that depend on the higher level. In this case, a hierarchical reasoning analysis can be used that takes the approach of recognizing and reasoning the human body, shoulders, arms, and hands in a hierarchical order, in order to track the movement of the fingers and recognize the "

According to one embodiment of the present invention, the reasoning unit 320 can generate an assertion value of an interaction type by applying an adaptive reasoning combination based on at least one of rule-based reasoning, Bayesian-based reasoning and fuzzy-based reasoning .

According to one embodiment of the present invention, rule-based reasoning can be a method of inferring results by predetermined logical rules. For example, rule-based reasoning uses a predetermined logical rule such as 'A' and 'B' in the implementation of the algorithm, and the result can be inferred by the logical rule.

According to one embodiment of the present invention, Bayesian-based reasoning is a method of statistical reasoning, which may be a method of deducing the posterior probability of the object through the prior probability of the object to be inferred and the additional observation of the object. Bayesian based reasoning can include temporal reasoning and probabilistic reasoning.

In Fig. 7, Bayesian integration through the steps of Bayesian-based reasoning is shown in detail.

According to one embodiment of the present invention, the prior knowledge may include a pre-probability and may include a plurality of recognition results generated by the plurality of recognizing units 330. [ The Bayesian theory can contain theorems that show the relationship between the prior probability and the posterior probability of two random variables. That is to say, the theorem of Bayesian theory can indicate how posterior probabilities are updated when new evidence is presented. A gain function (or a loss function) can provide a function of adding plus error and minus error. The measurement function can derive the result by combining the posterior probability value and the gain function value. Here, the result may include an assertion value of the interaction type. The measurement function may include a decision rule.

According to one embodiment of the present invention, in applying the adaptive reasoning combination based on Bayesian-based reasoning, the reasoning unit 320 may use at least one of a cue combination, hierarchical reasoning, and causal reasoning Can be used. A cue combination can be an inference method that yields results by combining revealed cues. Causal reasoning can be an inference method to derive a result through a causal relationship between situations based on current situations.

According to one embodiment of the present invention, the fuzzy-based reasoning can be an inference based on the fuzzy theory. In contrast to the fact that the logic circuit of a computer is driven by the absolute criterion of '0' and '1', fuzzy theory explores ways in which thinking, learning and self-development, It can be a branch of neural network. In fuzzy theory, in understanding the human emotions, such as non-deterministic information including probability, or ambiguous information, which is not correct judgment, it is necessary to develop a variety of application products in the field of automation in which complex and subtle information, It can be a theory that can be groundbreaking. For example, assuming that the average height of a Korean man is 170cm, according to the conventional theory, a person having an average height above the average can be defined as a large person. In other words, a person with a height of 169.5 cm is considered to be taller than a person with a height of 170.5 cm. However, the fuzzy theory can express the degree of magnitude such as 'a little bit small' or 'a little bit big' considering the median value. Therefore, fuzzy theory can enable various tasks that can not be represented by scientific theories that make determinations made by dichotomy so far. Human speech, behavior, thinking, or evaluation is highly subjective, and many limitations and problems can arise to be specified in concrete numbers. Fuzzy theory does not divide human behavior by dichotomy but introduces stochastic theory, so it can accept and understand ambiguous standards and expressions about human behavior.

The embodiments of the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specially designed and constructed for the present invention or may be those known to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.

Therefore, the spirit of the present invention should not be construed as being limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the following claims, I will say.

300: electronic device
310: Interaction type part
320: Reasoning section
330: Multiple identifiers
400: User Interface Engine
410: Interaction type hierarchy
420: Perceptual reasoning hierarchy
430: recognition algorithm layer

Claims (21)

A method for providing a user interface supporting interaction,
(a) the electronic device generating a plurality of recognition results using a plurality of recognition algorithms; And
(b) generating, by the electronic device, the assurance value of the interaction type indicating the semantic intensity of the interaction type by combining the plurality of recognition results according to the definition of the interaction type
≪ / RTI > The method according to claim 1,
(a0) the electronic device is configured to define the interaction type
≪ / RTI > The method according to claim 1,
Wherein the electronic device changes a combination of a plurality of recognition results when change information of the definition of the interaction type is obtained. The method according to claim 1,
In the step (b)
Wherein the electronic device generates the confidence value by quantitatively combining the plurality of recognition results. The method according to claim 1,
In the step (b)
Wherein the electronic device generates the confidence value by applying an inferential combination adaptive to the plurality of recognition results. 6. The method of claim 5,
In the step (b)
Wherein the electronic device generates the confidence value by applying an adaptive reasoning combination to the plurality of recognition results based on at least one of probabilistic analysis, structural analysis, and temporal analysis. 6. The method of claim 5,
In the step (b)
Wherein the electronic device generates the confidence value by applying the adaptive reasoning combination based on at least one of rule based reasoning, Bayesian based reasoning and fuzzy based reasoning. The method according to claim 1,
Wherein the plurality of recognition algorithms are used by the electronic device via a public interface module and the plurality of recognition results are provided for generation of the confidence value via the common interface module. The method according to claim 1,
Wherein the interaction type is a plurality, and the confidence value is generated for each of the plurality of interaction types. The method according to claim 1,
In the step (b)
Wherein the confidence value is generated for each frame of the image or for every frame period. A computer program for executing the method according to any one of claims 1 to 10. An electronic device for providing a user interface supporting interaction,
A plurality of recognizers for generating a plurality of recognition results using a plurality of recognition algorithms; And
And a reasoning unit for generating a confidence value of the interaction type indicating the intensity of the interaction type by combining the plurality of recognition results according to the definition of the interaction type,
≪ / RTI > 13. The method of claim 12,
An interaction type unit for defining the interaction type,
Lt; / RTI > 13. The method of claim 12,
When the change information of the definition of the interaction type is obtained,
Wherein the reasoning unit changes the combination of the plurality of recognition results according to a change of the definition of the interaction type. 13. The method of claim 12,
And the reasoning unit generates the confidence value by quantitatively combining the plurality of recognition results. 13. The method of claim 12,
Wherein the inference unit generates the confidence value by applying an inference combination adaptive to the plurality of recognition results. 17. The method of claim 16,
Wherein the inference unit generates the confidence value by applying an adaptive reasoning combination to the plurality of recognition results based on at least one of a probabilistic analysis, a structural analysis, and a temporal analysis. 17. The method of claim 16,
Wherein the inference unit generates the confidence value by applying the adaptive reasoning combination based on at least one of rule based reasoning, Bayesian based reasoning and fuzzy based reasoning. 13. The method of claim 12,
Wherein the plurality of recognition algorithms are used by the electronic device via a common interface module and the plurality of recognition results are provided for generation of the confidence value via the common interface module. 13. The method of claim 12,
Wherein the interaction type is a plurality, and the confidence value is generated for each of the plurality of interaction types. 13. The method of claim 12,
Wherein the confidence value is generated for each frame of the image or for every frame period.

Images