CN103809733B - Human-computer interaction system and method - Google Patents

Abstract

A human-computer interaction system and a human-computer interaction method are provided, the human-computer interaction system comprising: an image acquisition device for acquiring image data; a human-computer interaction processing device which determines an interactive operation that a user wants to perform according to a plurality of types of actions and gestures of the user detected from the image data; and the display device displays a display screen corresponding to the result of the interactive operation. The invention can utilize the combination of various motion detection modes to carry out human-computer interaction operation, thereby reducing the ambiguity of human-computer interaction operation identification and improving the accuracy of human-computer interaction operation under the condition of not needing an additional input device.

Description

Human-computer interaction system and method

technical field

The present invention relates to the field of computer vision and pattern recognition, more specifically, to a non-contact, natural long-distance human-computer interaction (HCI) system and method.

Background technique

The human-computer interaction method based on computer vision technology can visually obtain user input through various image acquisition and processing methods. The human-computer interaction method based on computer vision technology has become a hot topic in the new generation of human-computer interaction technology, especially in the human-computer interaction of leisure and entertainment, which has been widely used. In this interactive mode, the user can interact with the computer through the user's body posture, head posture, line of sight or human body movements, so that the user can be freed from the traditional keyboard, mouse, etc. input methods, and obtain unprecedented man-machine interaction. Interactive experience.

At present, a variety of human-computer interaction methods based on computer vision have been proposed. In one existing way of human-computer interaction, 3D objects can be created, modified and manipulated by using touch input and three-dimensional (3D) gesture input. In another approach, human gesture detection can be used to interact with the virtual user interface.

However, the types of motion detection utilized by existing human-computer interaction devices and methods are relatively single, generally requiring touch-based input devices and requiring users to memorize a large number of prescribed actions to perform the interaction. Due to the range of gestures, poses, and depth sensing, preprocessing or various manual operations are usually required, for example, various sensors need to be calibrated, interaction spaces are pre-defined, etc. This inconveniences the user. Therefore, there is a need for a human-computer interaction method that can utilize multiple motion detection methods and does not rely on additional input devices.

Contents of the invention

According to an aspect of the present invention, there is provided a human-computer interaction system, including: an image acquisition device for acquiring image data; a human-computer interaction processing device for processing various types of actions and gestures of the user detected from the image data The interactive operation that the user wants to perform is determined; the display device displays a display screen corresponding to the result of the interactive operation.

According to one aspect of the present invention, the human-computer interaction processing device includes: a motion detection module, which detects various types of actions and gestures of the user from image data; an interaction determination module, which detects various types of actions of the user according to the motion detection module and posture to determine the interactive operation that the user wants to perform, and send a corresponding display operation instruction to the display control module; the display control module controls the display device to display the corresponding interactive operation on the display screen according to the instruction determined by the interaction determination module.

According to one aspect of the present invention, the motion detection module includes: a gaze capture module, used to detect the user's gaze direction from the image data; a gesture tracking module, used to track and recognize gestures and actions of various parts of the user's body in the image data.

According to an aspect of the present invention, the sight-line capturing module determines the user's line-of-sight direction by detecting the pitch direction and the yaw direction of the user's head from the image data.

According to an aspect of the present invention, the gesture tracking module tracks and detects the nodes of the user's hand in the image data to determine the motion and gesture of the user's hand, and detects the nodes of the user's body skeleton to determine the gestures of various parts of the user's body.

According to an aspect of the present invention, the interaction determining module determines whether to start the interactive operation according to the user's gaze direction detected by the gaze capturing module and the gesture of the user's hand recognized by the gesture tracking module.

According to an aspect of the present invention, if it is determined that both the user's gaze direction and the user's hand pointing direction point to the display item on the display screen for more than a predetermined time, the interaction determination module determines to start an interactive operation on the display item.

According to an aspect of the present invention, if it is determined that neither the user's gaze direction nor the user's hand pointing direction points to the display item, the interaction determining module determines to stop the interactive operation on the display item.

According to an aspect of the present invention, when the user is close to the image acquisition device, the posture tracking module tracks and recognizes the user's finger movement to recognize the user's gesture, and when the user is far away from the image acquisition device, the posture tracking module tracks and recognizes the user's arm movement .

According to an aspect of the present invention, the human-computer interaction processing device further includes: a custom gesture registration module, configured to register an interactive operation command corresponding to a user-defined gesture action.

According to another aspect of the present invention, a human-computer interaction method is provided, including: acquiring image data; determining the interaction operation that the user wants to perform according to various types of actions and gestures of the user detected from the image data; displaying and The display screen corresponding to the result of the interactive operation.

According to another aspect of the present invention, the step of determining the interactive operation includes: detecting various types of actions and gestures of the user from the image data; determining the interactive operation to be performed according to the detected various types of actions and gestures of the user, And issue a display operation instruction corresponding to the interactive operation; control the display device to display the corresponding interactive operation on the display screen according to the determined instruction.

According to another aspect of the present invention, the step of detecting various types of actions and gestures of the user includes: detecting the gaze direction of the user from image data; tracking and recognizing gestures and actions of various parts of the user's body.

According to another aspect of the present invention, the user's line of sight direction is determined by detecting the pitch direction and the yaw direction of the user's head from the image data.

According to another aspect of the present invention, by tracking and detecting the nodes of the user's hand in the image data to determine the motion and gesture of the user's hand, and by detecting the nodes of the user's body skeleton from the image data to determine the shape of each part of the user's body Gesture action.

According to another aspect of the present invention, whether to start the interactive operation is determined according to the detected gaze direction of the user and the gesture of the user's hand recognized by the gesture tracking module.

According to another aspect of the present invention, if it is determined that both the user's gaze direction and the user's hand pointing direction point to a display item on the display screen for more than a predetermined time, it is determined to start an interactive operation on the display item.

According to another aspect of the present invention, if it is determined that neither the user's gaze direction nor the user's hand pointing direction points to the display item, it is determined to stop the interactive operation on the display item.

According to another aspect of the present invention, when the user is close to the image capturing device, the user's finger movement is tracked and recognized to recognize the user's gesture, and when the user is away from the image capturing device, the user's arm movement is recognized.

According to another aspect of the present invention, the step of determining the interaction operation further includes: determining the interaction operation corresponding to the registered user-defined gesture action.

Description of drawings

The above and other objects and features of the present invention will become more apparent from the following description in conjunction with the accompanying drawings exemplarily showing embodiments, in which:

FIG. 1 is a schematic diagram illustrating the interaction between a human-computer interaction system and a user according to an embodiment of the present invention;

2 is a structural block diagram showing a human-computer interaction processing device of a human-computer interaction system according to an embodiment of the present invention;

Fig. 3 is a schematic diagram showing a gesture of starting or stopping a human-computer interaction operation according to another embodiment of the present invention;

FIG. 4 is a flowchart illustrating a human-computer interaction method according to an embodiment of the present invention;

FIG. 5 is a flow chart illustrating menu operations performed by a human-computer interaction method according to an embodiment of the present invention;

FIG. 6 is a flow chart illustrating the interactive operation of a 3D display target according to a human-computer interaction method according to an embodiment of the present invention;

Fig. 7 is a flow chart showing a handwriting operation performed by a human-computer interaction method according to an embodiment of the present invention.

Detailed ways

Reference will now be made in detail to embodiments of the invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments are described below in order to explain the present invention by referring to the figures.

Fig. 1 is a schematic diagram illustrating interaction between a human-computer interaction system and a user according to an embodiment of the present invention.

As shown in FIG. 1 , the human-computer interaction system according to the embodiment of the present invention includes an image acquisition device 100 , a human-computer interaction processing device 200 and a display device 300 . The image acquisition device 100 is used to acquire image data, and the image data may have depth features and color features. The image acquisition device 100 may be a device capable of capturing depth images, for example, a depth camera.

The human-computer interaction processing device 200 is configured to analyze the image data acquired by the image acquiring device 100, so as to recognize the user's gesture and action and analyze the user's gesture and action. Then, the human-computer interaction processing device 200 controls the display device 300 to perform corresponding display according to the analysis result. The display device 300 may be a device such as a television (TV) or a projector.

Here, as shown in FIG. 1 , the human-computer interaction processing device 200 may determine the interaction operation that the user wants to perform according to various types of detected actions and gestures of the user. For example, a user may point a finger at a specific object (OBJ2) among a plurality of objects (eg, OBJ1, OBJ2, and OBJ3 shown in FIG. object to start interacting. That is, the human-computer interaction processing device 200 can detect the user's gaze direction, gestures, and actions and postures of various parts of the body. The user can also operate a certain displayed object by moving a finger, for example, change the display position of the object. At the same time, the user can also move a certain part of the body (for example, an arm) or move the whole body to input the interactive operation. It should be understood that although the image acquisition device 100 , the human-computer interaction processing device 200 and the display device 300 are shown as separate devices, these three devices may also be arbitrarily combined into one or two devices. For example, the image acquisition device 100 and the human-computer interaction processing device 200 can be implemented in one device.

The structure of the human-computer interaction processing device 200 in the human-computer interaction system according to the embodiment of the present invention will be described in detail below with reference to FIG. 2 .

As shown in FIG. 2 , the human-computer interaction processing device 200 according to the embodiment of the present invention includes a motion detection module 210 , an interaction determination module 220 and a display control module 230 .

The motion detection module 210 is used to detect various types of actions of the user and determine the gesture of the user. For example, the motion detection module 210 may detect and determine motion of the user's gaze direction, motion of body parts, gestures, and body gestures. The interaction determination module 220 may determine an interaction operation to be performed according to various types of actions and gestures of the user detected by the motion detection module 210 . The operation process of the motion detection module 210 will be described in detail below.

According to an embodiment of the present invention, the motion detection module 210 may include a gaze capture module 211 and a gesture tracking module 213 .

Wherein, the line-of-sight capturing module 211 is used to obtain the line-of-sight direction of the user from the image data. The user's gaze direction may be obtained by detecting the user's head posture from image data. The posture of the head is mainly reflected by head pitch and head yaw. Correspondingly, the pitch angle and yaw angle of the head can be estimated respectively in the head area in the depth image, so as to synthesize the corresponding head pose based on the pitch angle and yaw angle, and thus obtain the user's line of sight direction.

The gesture tracking module 213 is used for tracking and recognizing gestures and actions of various parts of the user's body. For example, the gesture tracking module 213 can track and recognize the user's pointing direction and finger motion from the acquired image data. The gesture tracking module 213 can track the trajectory and speed of the hand. In addition, the posture tracking module 213 can also track and recognize movements of various parts of the user's body (eg, arms). Preferably, in the mode where the user is close to the image acquisition device 100, the posture tracking module 213 can track the nodes of the user's hand through dense and reliable image data, so as to determine the pointing direction and motion (that is, gesture) of the user's finger. ). While in the mode where the user is far away from the image acquisition device 100, since the acquired image data is relatively rough, with more noise and the hand area is small, the gesture tracking module 213 can track the user's upper arm (that is, the wrist node) by tracking the skeletal nodes of the human body. and the bone between the elbow node) to track and recognize the user's arm pointing direction and movement.

To this end, according to an embodiment of the present invention, the gesture tracking module 213 may recognize and track the movement of the user's hand based on the skin color feature and/or the 3D feature. Specifically, the pose tracking module 213 may include a classifier trained based on skin color or 3D features. For the case of using a skin color classifier, a probabilistic model (eg, Gaussian Mixture Model (GMM)) can be used to distinguish whether a possible pixel belongs to a hand through the color distribution of the hand skin. For depth features, depth comparison features can be generated as described in "Real-Time Human Pose Recognition in Parts from Single Depth Images. JamieShotton et al. In CVPR 2011", or local depth blocks (small rectangular blocks) can be compared with existing Blocks on the hand model are compared and the similarity is measured. Then, combining different color features and depth features, a general classifier (such as Random Forest or AdaBoosting decision tree) can be used to perform a classification task to determine the hands in the image data. Then, by detecting the hand frame by frame, the pose tracking module 213 can track and calculate the motion trajectory/velocity of the hand to locate the hand in the 2D image and 3D spatial domain. In particular, the position of the hand joints can be tracked by comparing the depth data with the 3D hand model. However, if the hand is far away from the image acquisition device 100, when the hand area in the image data is smaller than a predetermined threshold, considering data reliability, the arm motion can be determined by tracking the user's body skeleton.

The interaction determination module 220 may determine an interaction operation to be performed according to various actions of the user detected by the motion detection module 210 . For example, the interaction determining module 220 may determine whether to enter an interactive operation gesture according to the user's gaze direction determined by the gesture tracking module 211 and the user's pointing direction determined by the gesture tracking module 213, and determine whether to perform interactive operation. That is to say, the interaction determination module 220 may determine the start or end of the interaction operation according to the user's gaze direction and the user's pointing direction. Specifically, when the gesture tracking module 211 determines that the user's gaze direction and the gesture tracking module 213 determine the user's pointing direction both point to a certain target displayed on the display device 300 (that is, the intersection of the gaze direction and the pointing direction of the finger has When a specific display target) exceeds a predetermined time, the interaction determination module 220 may determine that the user wants to start interacting to operate the display target. During the operation on the display object, the interaction determination module 220 determines whether at least one of the user's line of sight and pointing direction is still on the display object. When neither the user's line of sight nor the pointing direction remains on the target, the interaction determination module 220 may determine that the user stops the interactive operation with the displayed target. Through the above manner, it can be more accurately determined whether the user starts or ends the interactive operation, thereby improving the accuracy of the interactive operation.

It should be understood that the above is only an example of determining whether to start or end the interactive operation state according to the detected actions and gestures of the user. Whether to start or end the interactive operation state can also be determined according to other preset manners. For example, the interactive operation gesture may be started according to the user's gaze direction and a predetermined gesture. As shown in FIG. 3 , when the motion detection module 210 determines from the image data that the user's fingers are spread open and the line of sight is directed to a specific item on the display screen of the display device 300, the interaction determination module 220 may determine that the user wants to select the specific item on the display screen of the display device 300. Items interact. Next, when the motion detection module 210 determines that the user's fingers are pinched together and the hand starts to move, the interaction determination module 220 may determine that the user wants to drag a specific item. If the motion detection module 210 determines that the user's hand is clenched into a fist, the interaction determination module 220 may determine that the user wants to stop the interactive operation.

After entering the interactive operation state, the interaction determination module 220 also determines the interactive operation that the user wants to perform according to the user's actions and gestures. According to an embodiment of the present invention, the interaction determining module 220 may determine the interactive operation of moving the pointer according to the pointing direction of the user's hand. According to the pointing direction of the user's hand determined by the gesture tracking module 213, the interaction determining module 220 can calculate the intersection point of the pointing direction and the display screen, so as to obtain the position of the pointer on the display screen. When the user's hand moves, the interaction determination module 220 can issue a corresponding command, instructing the display control module 230 to control the display of the display device 300, so that the pointer also moves on the screen with the movement of the hand.

According to an embodiment of the present invention, the interaction determining module 220 may also determine the interactive operation of the button according to the user's hand motion determined by the gesture tracking module 213 . According to the pointing direction of the user's hand determined by the gesture tracking module 213, the interaction determining module 220 can calculate the intersection point of the pointing direction and the display screen, and if there is a display item such as a button at this position, the interaction determining module 220 can determine that the user presses Press the button. Alternatively, if the gesture tracking module 213 determines that the user's finger/fist is moving rapidly along its pointed direction, the interaction determination module 220 determines that the button is pressed.

It should be understood that here are just a few examples where the interaction determination module 220 determines the interaction operation that the user wants to perform according to the gaze direction determined by the gaze tracking module 210 and the user's gesture action determined by the gesture tracking module 213 . However, those skilled in the art should understand that the interactive operation of the present invention is not limited thereto. More interactive operations can also be performed according to the user's gestures and/or the direction of the user's line of sight, such as dragging the display target by moving the hand, rotating the display target, clicking or double-clicking the display target through finger movements, and the like.

In addition, according to the embodiment of the present invention, the user can also customize the interactive operation corresponding to a specific action gesture. To this end, the human-computer interaction processing device 200 may further include a custom gesture registration module (not shown), configured to register interactive operations corresponding to user-defined gestures and actions. The custom gesture registration module may have a database for mapping recorded gestures and actions to corresponding interactive manipulation commands. For example, in the case of 2D or 3D object display, the 2D or 3D displayed object can be reduced or enlarged by tracking the movement directions of the two hands. In particular, to register new gestures, the custom gesture registration module tests the reproducibility and ambiguity of user-defined gestures and returns a reliability score to indicate whether the user-defined interaction commands are valid.

After the interaction determination module 220 determines the interaction operation that the user wants to perform, the interaction determination module 220 sends a corresponding instruction to the display control module 230, and the display control module 230 controls the display device 300 to display the corresponding interaction operation on the display screen according to the instruction. For example, the display device 300 may be controlled to display a screen image of operations such as a pointer being moved, a corresponding display item being moved, a button being pressed, and the like.

The specific process of the human-computer interaction method according to the embodiment of the present invention will be described below with reference to FIG. 4 .

As shown in FIG. 4 , in step S410 , image data is firstly acquired by the image acquisition device 100 .

Next, in step S420, the human-computer interaction processing device 200 analyzes various types of user gestures and actions in the image data acquired by the image acquisition device 100 to determine whether to enter the interactive operation state and the interactive operation that the user wants to perform. Here, for example, the human-computer interaction processing device 200 may detect and recognize the user's gaze direction and the actions and gestures of various parts of the human body from the image data to determine the interactive operation the user wants to perform. According to this embodiment, the human-computer interaction processing device 200 may determine whether to enter the interactive operation state according to the detected gaze direction and the user's pointing direction. Specifically, when the human-computer interaction processing device 200 determines from the image data that the user's gaze direction and hand pointing direction point to a certain display item displayed on the display screen of the display device 300 for more than a predetermined time, the human-computer interaction processing The device 200 enters the interactive operation state, and determines the interactive operation to be performed on the display target according to the user's subsequent gesture action.

Then, in step S430, the display device 300 is controlled to display the corresponding display screen or update the display screen according to the determined interactive operation. For example, a position where the user wants to move a displayed pointer, drag a displayed item, click a displayed item, double-click a displayed item, etc. may be determined according to the pointing direction of the user's hand.

In step S420, if the human-computer interaction processing device 200 determines that both the pointing direction and the line of sight direction of the user have left the display target during the execution of the interactive operation, then it is determined that the user wants to stop the interactive operation on the display target, and the display stops. The display screen on which the target operates. It should be noted that other ways can also be used to determine whether the user wants to stop the interactive operation. For example, the interactive operation may be stopped according to a specific gesture of the user (clenching a fist as described above).

The schematic flow of performing various interactive operations by using the human-computer interaction method according to the present invention will be described below with reference to FIGS. 5-7 .

FIG. 5 shows a flow chart of menu operations performed by a human-computer interaction method according to an embodiment of the present invention.

In the embodiment of FIG. 5 , it is assumed that the preset menu is displayed on the display screen of the display device 300 , and the preset menu includes several items for the user to perform interactive operations.

In step S510, when the human body posture detected from the captured image data shows that both the pointing direction of the user's hand and the direction of sight point to a specific menu item on the display screen, it is determined to enter the interactive operation state for the menu.

Next, in step S520, the trajectory and speed of the user's hand can be tracked to determine the user's hand motion and gesture, and the interactive operation that the user wants to perform can be determined according to the hand motion and gesture. For example, the interactive operation of the mouse can be simulated according to the movement of the user's hand. When it is determined that the user's index finger makes a click action, a specific item of the menu in the direction pointed by the finger may be selected. When it is determined that the user's middle finger performs a click action, content corresponding to the right mouse button action may be displayed, for example, additional menu options related to the item may be displayed. Then, in step S530, the display device is controlled to display or update the menu content corresponding to the determined interactive operation.

Fig. 6 is a flowchart of an operation of displaying a target in 3D according to a human-computer interaction method according to an embodiment of the present invention. Here, the display device 300 is a display device that can display 3D content.

First, in step S610, when the human body posture detected from the captured image data shows that the pointing direction of the user's hand and the direction of sight point to a specific 3D display object on the display screen, it is determined to enter the interactive operation on the 3D display object state. Next, in step S620, the trajectory and speed of the user's hand can be tracked to determine the user's hand motion and gesture, and the interactive operation that the user wants to perform can be determined according to the hand motion and gesture. For example, the 3D display object at the intersection of the pointing direction of the hand and the line of sight direction can be picked up, and the 3D display object can be moved according to the movement of the hand. In addition, the selected 3D display target can also be dragged, enlarged or reduced according to the movement of the hand. Finally, in step S630, the display device is controlled to re-render the 3D display object after the interactive operation according to the determined interactive operation.

Fig. 7 is a flowchart of a text input operation performed by a human-computer interaction method according to an embodiment of the present invention. Here, it is assumed that a predetermined area on the display screen displayed by the display device 300 can be used as a text input area.

First, in step S710, when the human body posture detected from the captured image data shows that the pointing direction of the user's hand and the direction of sight both point to the handwriting input area on the display screen, it is determined to enter the interactive operation state of handwriting input. Next, in step S720, the trajectory and speed of the user's hand may be tracked, and the text that the user wants to input may be determined according to the trajectory of the user's hand. The text that the user wants to input may be determined according to a learning-based recognition method, and the text may be interpreted as a corresponding interactive operation command. Finally, in step S730, the display device is controlled to display a display screen of the result after the execution of the interactive operation command.

It should be understood that although the above embodiments determine whether to start or end the interactive operation and the user's subsequent interactive operation according to the direction of sight and the pointing direction of the hand, the present invention is not limited thereto. Whether to start or end an interactive operation and subsequent interactive operations may be determined based on detecting a combination of other types of motion detection.

According to the present invention, a combination of multiple motion detection methods can be used for human-computer interaction, thereby reducing the ambiguity of human-computer interaction recognition and improving The accuracy of human-computer interaction operation. For example, without using the touch screen input device, the interactive operation of zooming in and zooming out the displayed object can be realized. In this way, the motion detection method of the computer vision technology is fully utilized, and a better interactive operation experience is brought to the user.

While the invention has been shown and described with reference to several exemplary embodiments thereof, those skilled in the art will understand that, without departing from the spirit and scope of the invention as defined in the claims and their equivalents, other modifications may be made. Various changes were made in form and detail.

Claims (14)

1. a kind of man-machine interactive system, including：

Image acquisition equipment, for obtaining image data；

Sight capture module determines user by detecting pitch orientation and the deflection direction on the head of user from image data Direction of visual lines；

Posture tracing module, the instruction direction of the hand for user to be tracked and identified in image data；

Interaction determining module, is directed to display items to determine interaction based on the instruction direction of the direction of visual lines of user and the hand of user The beginning of operation；

Wherein, the interactive determining module：

At least one of the instruction direction of hand of direction of visual lines and user in response to determining user still points to display items to protect Interactive operation is held,

Display items are not directed toward to stop interactive operation in the instruction direction of the hand of direction of visual lines and user in response to determining user.

2. man-machine interactive system as described in claim 1, wherein posture tracing module be additionally operable in image data tracking and Identify the posture and action of user's body each section.

3. man-machine interactive system as claimed in claim 2, wherein use is tracked in image data and detected to posture tracing module The node of the hand at family detects the body bone node of user to determine user's body to determine movement and the gesture of the hand of user The posture of each section acts.

4. man-machine interactive system as claimed in claim 3, wherein appearance of the interaction determining module also according to user's body each section State determines beginning interactive operation with action.

5. man-machine interactive system as claimed in claim 4, wherein the use that interaction determining module is detected according to sight capture module Family direction of visual lines and posture tracing module identification user hand action come determine whether start interactive operation.

6. man-machine interactive system as claimed in claim 2, wherein when user is close to image acquisition equipment, posture tracks mould Block is tracked and the finger movement of identification user is to identify the gesture of user, when user is far from image acquisition equipment, posture tracking Module is tracked and the action of the arm of identification user.

7. man-machine interactive system as described in claim 1, further includes：

It shows equipment, shows display screen corresponding with the result of interactive operation,

Wherein, if it is determined that the display items that the instruction direction of the direction of visual lines of user and the hand of user is directed on display screen are super The predetermined time is spent, then starts interactive operation.

8. man-machine interactive system as described in claim 1, further includes：

Self-defined posture registration module, for registering interaction operation command corresponding with user-defined gesture actions.

9. a kind of man-machine interaction method, including：

Obtain image data；

The direction of visual lines of user is determined by detecting pitch orientation and the deflection direction on the head of user from image data；

The instruction direction of the hand of user is tracked and identified in image data；

It is directed to display items based on the instruction direction of the direction of visual lines of user and the hand of user to determine the beginning of interactive operation；

Wherein, in response to determining that the direction of visual lines of user and at least one of the instruction direction of hand of user still point to display items Interactive operation is kept, is stopped in response to determining that display items are not directed toward in the instruction direction of the direction of visual lines of user and the hand of user Only interactive operation.

10. man-machine interaction method as claimed in claim 9, wherein by the hand that user is tracked and detected in image data Node to determine movement and the gesture of the hand of user, and by detecting the body bone node of user from image data with true Determine the posture action of user's body each section.

11. man-machine interaction method as claimed in claim 9, wherein tracked according to the direction of visual lines of the user of detection and posture Module identification user hand action come determine whether start interactive operation.

12. man-machine interaction method as claimed in claim 9, wherein if it is determined that the hand of the direction of visual lines and user of user It indicates that the display items that direction is directed on display screen are more than the predetermined time, then starts interactive operation.

13. man-machine interaction method as claimed in claim 9, wherein when user is close to image acquisition equipment, tracking and identification The finger movement of user, when user is far from image acquisition equipment, identifies the action of the arm of user to identify the gesture of user.

14. man-machine interaction method as claimed in claim 9, wherein the step of determining interactive operation further include：It determines and registers The corresponding interactive operation of user-defined gesture actions.