CN111652155A - A method and system for recognizing human motion intention - Google Patents

Abstract

The invention discloses a method and a system for identifying human motion intention. The method comprises the following steps: acquiring a motion coordinate in the motion of a human body; determining coordinates of a sight line attention point in human motion through a wearable glasses type eye movement instrument; recognizing the coordinates of obstacles in a visual field scene in the motion of a human body through a trained convolutional neural network model; and predicting the human body movement yaw angle information through a trained recurrent neural network model according to the movement coordinate, the attention point coordinate and the obstacle coordinate. The exoskeleton robot can quickly and accurately identify the walking movement direction of the human body so as to provide accurate wearer movement information for the exoskeleton robot.

Description

A method and system for recognizing human motion intention

technical field

The present invention relates to the technical field of computer pattern recognition, in particular to a method and system for recognizing human motion intention.

Background technique

With the continuous advancement of science and technology, the proportion of machines participating in human life is increasing. Mechanical exoskeletons can be used in medical treatment, rehabilitation training, military assistance and other scenarios. Powered exoskeletons can improve the ability of individual soldiers to fight, and power exoskeletons can compensate for the patient's walking ability and assist patients in rehabilitation training.

To achieve efficient human-machine collaborative work, strong human-machine interaction capabilities and intelligent control are required. In traditional human-computer interaction methods, the problems of redundant robot motion, accumulation of motion errors in human-computer interaction systems, and poor user interaction experience have not been effectively solved. As a participant in the human-computer interaction system, the human control system itself is the most advanced, complex and intelligent system on the earth. In order to improve the ability of human-computer interaction, the most effective way is to imitate human beings and organically integrate multiple disciplines such as biology, control technology, sensor data fusion, and machine learning to obtain the best platform for robots to be intelligent and friendly. The establishment of an intelligent human-computer interaction strategy can be divided into the following three stages: human motion intention estimation, robot assist control and stability control. How to qualitatively and quantitatively capture and analyze human motion intention and map it to the input of the compliance control system quickly and accurately, and then complete the semi-automatic control strategy based on the exoskeleton wearing operator-based control model and the active control model based on the human-robot coordinated control model The intelligent control strategy is the premise and foundation to provide the wearer with natural comfort, reasonable health and friendly presence. In the process of the lower limb exoskeleton robot interacting with the wearer to assist the movement, the perception system has always been a research difficulty to capture the human movement intention in real time, quickly and accurately.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method and system for recognizing human body motion intention, so as to obtain the human body motion intention accurately and quickly.

For achieving the above object, the present invention provides the following scheme:

A method for identifying human motion intention, the method comprising:

Get the motion coordinates in human motion;

Determine the coordinates of the attention point of sight in human movement through a wearable eye-tracking device;

Identify the coordinates of obstacles in the visual field of human motion through the trained convolutional neural network model;

According to the motion coordinates, the attention point coordinates and the obstacle coordinates, the human body motion yaw angle information is predicted through the trained recurrent neural network model.

Optionally, the acquiring motion coordinates in the motion of the human body specifically includes:

Obtain video information from a monocular camera worn on the human body;

Obtain the motion information of the IMU sensor worn on the human body;

Using the visual inertial odometry scheme, the video information and the motion information are processed to obtain motion coordinates in the motion of the human body.

Optionally, the determining the coordinates of the attention point of sight in the human body movement by using a wearable eye-tracking device specifically includes:

Obtain the video information of the pupil of the human eye through the wearable glasses eye tracker;

According to the video information of the pupil of the human eye, the coordinates of the attention point of the line of sight in the movement of the human body are determined.

Optionally, identifying the coordinates of obstacles in the visual field scene in human motion through the trained convolutional neural network model specifically includes:

Obtain the video information of the visual field scene in the human body movement through the wearable glasses-type eye tracker;

Identify each frame of image in the video information of the visual field by using the trained convolutional neural network model, and determine the obstacles in the image;

The obstacles in each frame of image are selected by a rectangular frame, and the coordinates of the obstacles in the image are obtained.

Optionally, the method further includes: using a spline interpolation method to synchronize the motion coordinates, the focus point coordinates and the obstacle coordinates into signals at equal intervals.

The present invention also provides a system for identifying human motion intentions, the system comprising:

The motion coordinate acquisition module is used to acquire motion coordinates in human motion;

The attention point coordinate determination module is used to determine the coordinates of the line of sight attention point in the human body movement through the wearable glasses eye tracker;

The obstacle coordinate recognition module is used to identify the obstacle coordinates in the visual field of human motion through the trained convolutional neural network model;

The prediction module is used for predicting the yaw angle information of human body motion through the trained recurrent neural network model according to the motion coordinates, the attention point coordinates and the obstacle coordinates.

Optionally, the motion coordinate acquisition module specifically includes:

a first video information acquisition unit, used for acquiring video information in a monocular camera worn on the human body;

A motion information acquisition unit, used to acquire motion information of the IMU sensor worn on the human body;

The processing unit is configured to use the visual inertial odometry scheme to process the video information and the motion information to obtain motion coordinates in the motion of the human body.

Optionally, the coordinate determination module of the point of interest specifically includes:

a second video information acquisition unit, configured to acquire video information of the pupil of the human eye through the wearable eye-tracker;

The attention point coordinate determination unit is used for determining the coordinates of the line of sight attention point in the motion of the human body according to the video information of the pupil of the human eye.

Optionally, the obstacle coordinate identification module specifically includes:

a third video information acquisition unit, configured to acquire video information of visual field scenes in human motion through the wearable glasses eye tracker;

The obstacle identification unit is used to identify each frame of image in the video information of the visual field through the trained convolutional neural network model, and determine the obstacle in the image;

The coordinate determination unit is used to frame the obstacles in each frame of images through a rectangular frame to obtain the coordinates of the obstacles in the image.

Optionally, also include:

The interpolation module is used for synchronizing the motion coordinates, the attention point coordinates and the obstacle coordinates into signals at equal intervals by using a spline interpolation method.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

(1) In the present invention, a module integrated with a wearable monocular camera and an IMU is used, and the visual inertial odometry solution is adopted, which can realize the positioning of the human body itself in the space environment without the need for an external positioning system, and the positioning information can be used to solve the human body. Yaw angle information during motion;

(2) In the present invention, the eye tracker is used to obtain the interest points that the human eye pays attention to in real time. Through this data, the wearer's degree of attention to objects in the space environment can be analyzed. Combined with the coordinate points of the identified objects described in the invention, wherein It can analyze the influence of objects on the walking direction of the human body;

(3) In the present invention, the cyclic neural network is used as a network model that integrates various information. The focus point can predict the yaw angle information of the person at the next moment, and obtain the human motion intention accurately and quickly.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative labor.

1 is a flowchart of a method for recognizing human motion intention according to an embodiment of the present invention;

2 is a schematic structural diagram of an LSTM cyclic neural network according to an embodiment of the present invention;

FIG. 3 is a structural block diagram of a system for recognizing human motion intention according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The purpose of the present invention is to provide a method and system for recognizing human body motion intention, so as to obtain the human body motion intention accurately and quickly.

In order to make the above objects, features and advantages of the present invention more clearly understood, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

As shown in Figure 1, a method for identifying human motion intention includes the following steps:

Step 101: Obtain motion coordinates in human body motion. Specifically, the video information in the monocular camera worn on the human body is obtained; the motion information of the IMU sensor worn on the human body is obtained; the visual inertial mileage scheme is used to process the video information and the motion information to obtain the human body Motion coordinates in motion.

In the example of the present invention, VINS-Mono is selected as the solution of the visual inertial odometer. First, the Zhang Zhengyou marking method is used to obtain the internal and external parameters of the camera. The IMU provides the drift and random error of the IMU when it leaves the factory, and uses the ORB method to identify the image corners. , the K nearest neighbor method is used to achieve the matching between the feature points of the two frames, the N-point perspective method is used to solve the rotation and translation of the camera, the pose of the camera is estimated, the IMU kinematics is used to calculate the IMU drift error, and the IMU pre-integration method is used to define the state The vector is all camera states in the sliding window, the change parameters from the camera to the IMU, the inverse depth of all 3D points, and the observation points are the observed matching feature points. The graph optimization method is used to solve the maximum a posteriori estimation of the observation points for the state vector, Then, the coordinate points of the human body in the space environment can be measured more accurately.

Step 102 : Determine the coordinates of the attention point of sight in the movement of the human body through the wearable eye-tracking device. Specifically, the video information of the through-hole of the human eye is obtained through the wearable eye-tracking device; the coordinates of the attention point of the line of sight in the movement of the human body are determined according to the video information of the pupil of the human eye.

A wearable eye-tracking device with glasses is used. After the human body wears the eye-tracking device, the video of the surrounding environment from the angle of the human eye can be recorded through the eye-tracking device. The video information of the eye pupil, according to the built-in algorithm of the eye tracker, to solve the plane two-dimensional pixel coordinate point (x, y) that the wearer's eyes in the surrounding video are looking at and the eye tracker device as the origin. The three-dimensional coordinate point (X, Y, Z) in the three-dimensional coordinate system.

Step 103: Recognize the coordinates of obstacles in the visual field scene during human motion through the trained convolutional neural network model. Specifically, the video information of the visual field scene in human motion is obtained through the wearable eye-tracking device; the trained convolutional neural network model is used to identify each frame of the image in the video information of the visual field scene, and determine the visual field scene video information in the image. The obstacles in each frame are selected by a rectangular frame, and the coordinates of the obstacles in the image are obtained.

The glasses-type eye tracker can record the video of the surrounding environment from the angle of the human eye, and input the video stream to the convolutional neural network for target detection. The convolutional neural network adopts the existing open source solution and is trained on the open source target detection data set in advance, and can identify pre-set label objects in the environment, such as human body, chair, monitor, etc. The video is divided into single-frame pictures, and each frame in the video is input to the convolutional neural network for identification, and the plane position of the label object in each frame of image is marked with a rectangular frame, and the center coordinates of the identified rectangular frame can be obtained ( x, y) and the height h and width w of the rectangular frame, as the position coordinates (x, y, w, h) of the object observed by the human eye.

Step 104: Predict the yaw angle information of human body movement through the trained recurrent neural network model according to the motion coordinates, the attention point coordinates and the obstacle coordinates.

The recurrent neural network is a long and short-term memory network model. The network inputs the coordinate vector of the human eye gaze point, the coordinate vector of the obstacle object (x, y, w, h), and the motion coordinate vector, and integrates the above vector coordinates into a vector as the input vector, Using the attitude coordinate information, the variation of the yaw angle at the next moment and the current moment is obtained as the label data corresponding to the current input vector, and input to the long and short-term memory network model. Training time The short-term memory network sets the step size to 3, that is, when training the current time T, the fully connected layer vector that has been output in the network by combining the historical positions and attitude coordinates of T-1 and T-2 times is passed. The network at the current moment jointly calculates a new output, and outputs the yaw angle value, which is used to predict the direction of human movement.

In the example of the present invention, the cyclic neural network is selected as a long-short-term memory LSTM network, and FIG. 2 is its network structure. The network is equipped with a memory gate and a forgetting gate, which can eliminate the gradient disappearance problem of the recurrent neural network, and has the ability to memorize long-term information. The synchronized signal is input into the bidirectional long-term and short-term memory network, and after network processing, the information of the rotation angle required by the trajectory movement at the current moment is output.

The key to LSTM is the memory block, which mainly includes three gates (forget gate, inputgate, output gate) and a memory unit (cell). The horizontal line above the box is called the cell state. The network update relationship of LSTM is as follows:

i _t =α(W _xi x _t +W _hi h _t-1 +W _ci c _t-1 +b _i )

f _t =α(W _xf x _t +W _hf h _t-1 +W _cf c _t-1 +b _f )

o _t =α(W _xo x _t +W _ho h _t-1 +W _co c _t-1 +b _o )

c _t =f _t *c _t-1 +i _t *tanh(W _xc x _t +W _hc h _t-1 +b _c )

h _t =o _t *tanh(c _t )

where i _t is the input gate, f _t is the forget gate, o _t is the output gate, c _t is the cell state, and h _t is the output of the t state.

The signals input to the cyclic neural network model have different frequencies and different phases, and need to be processed. The spline interpolation method is used to synchronize the above information into equally spaced signals.

As shown in FIG. 3 , the present invention also provides a system for recognizing human motion intention, the system comprising:

The motion coordinate acquisition module 301 is used for acquiring motion coordinates in human body motion.

The motion coordinate acquisition module 301 specifically includes:

a first video information acquisition unit, used for acquiring video information in a monocular camera worn on the human body;

A motion information acquisition unit, used to acquire motion information of the IMU sensor worn on the human body;

The processing unit is configured to use the visual inertial odometry scheme to process the video information and the motion information to obtain motion coordinates in the motion of the human body.

The attention point coordinate determination module 302 is used for determining the coordinates of the line of sight attention point in the human body movement through the wearable eye-tracker with glasses.

The attention point coordinate determination module 302 specifically includes:

a second video information acquisition unit, configured to acquire video information of the pupil of the human eye through the wearable eye-tracker;

The attention point coordinate determination unit is used for determining the coordinates of the line of sight attention point in the motion of the human body according to the video information of the pupil of the human eye.

The obstacle coordinate identification module 303 is configured to identify the obstacle coordinates in the visual field scene during human motion through the trained convolutional neural network model.

The obstacle coordinate identification module 303 specifically includes:

a third video information acquisition unit, configured to acquire video information of visual field scenes in human motion through the wearable glasses eye tracker;

The obstacle identification unit is used to identify each frame of image in the video information of the visual field through the trained convolutional neural network model, and determine the obstacle in the image;

The coordinate determination unit is used to frame the obstacles in each frame of images through a rectangular frame to obtain the coordinates of the obstacles in the image.

The prediction module 304 is configured to predict the yaw angle information of the human body movement through the trained recurrent neural network model according to the movement coordinates, the attention point coordinates and the obstacle coordinates.

The system also includes:

The interpolation module is used for synchronizing the motion coordinates, the attention point coordinates and the obstacle coordinates into signals at equal intervals by using a spline interpolation method.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

In this paper, specific examples are used to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present invention; meanwhile, for those skilled in the art, according to the present invention There will be changes in the specific implementation and application scope. In conclusion, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. a recognition method of human body motion intention, is characterized in that, described method comprises: Get the motion coordinates in human motion; Determine the coordinates of the attention point of sight in human movement through a wearable eye-tracking device; Identify the coordinates of obstacles in the visual field of human motion through the trained convolutional neural network model; According to the motion coordinates, the attention point coordinates and the obstacle coordinates, the human body motion yaw angle information is predicted through the trained recurrent neural network model. 2. the identification method of human body motion intention according to claim 1, is characterized in that, described obtaining the motion coordinates in human body motion, specifically comprises: Obtain video information from a monocular camera worn on the human body; Obtain the motion information of the IMU sensor worn on the human body; Using the visual inertial odometry scheme, the video information and the motion information are processed to obtain motion coordinates in the motion of the human body. 3. The method for recognizing human motion intention according to claim 1, characterized in that, determining the coordinates of the attention point of sight in human motion by a wearable eye-tracking device, specifically comprising: Obtain the video information of the pupil of the human eye through the wearable glasses eye tracker; According to the video information of the pupil of the human eye, the coordinates of the attention point of the line of sight in the movement of the human body are determined. 4. the identification method of human body motion intention according to claim 1, is characterized in that, described by the trained convolutional neural network model identifying the obstacle coordinates in the visual field scene in the human body motion, specifically comprises: Obtain the video information of the visual field scene in the human body movement through the wearable glasses-type eye tracker; Identify each frame of image in the video information of the visual field by using the trained convolutional neural network model, and determine the obstacles in the image; The obstacles in each frame of image are selected by a rectangular frame, and the coordinates of the obstacles in the image are obtained. 5 . The method for recognizing human motion intention according to claim 1 , further comprising: using a spline interpolation method to synchronize the motion coordinates, the coordinates of the point of interest, and the coordinates of the obstacle into equally spaced signals. 6 . 6. A recognition system for human body motion intention, wherein the system comprises: A motion coordinate acquisition module, used to acquire motion coordinates in human motion; The attention point coordinate determination module is used to determine the coordinates of the line of sight attention point in the human body movement through the wearable glasses eye tracker; The obstacle coordinate recognition module is used to identify the obstacle coordinates in the visual field of human motion through the trained convolutional neural network model; The prediction module is used for predicting the yaw angle information of human body motion through the trained recurrent neural network model according to the motion coordinates, the attention point coordinates and the obstacle coordinates. 7. The recognition system of human body motion intention according to claim 6, wherein the motion coordinate acquisition module specifically comprises: a first video information acquisition unit, used for acquiring video information in a monocular camera worn on the human body; A motion information acquisition unit, used to acquire motion information of the IMU sensor worn on the human body; The processing unit is configured to use the visual inertial odometry scheme to process the video information and the motion information to obtain motion coordinates in the motion of the human body. 8. The recognition system of human body motion intention according to claim 6, wherein the coordinate determination module of the point of interest specifically comprises: a second video information acquisition unit, configured to acquire video information of the pupil of the human eye through the wearable eye-tracker; The attention point coordinate determination unit is used for determining the coordinates of the line of sight attention point in the motion of the human body according to the video information of the pupil of the human eye. 9. The recognition system of human motion intention according to claim 6, wherein the obstacle coordinate recognition module specifically comprises: a third video information acquisition unit, configured to acquire video information of visual field scenes in human motion through the wearable glasses eye tracker; The obstacle identification unit is used to identify each frame of image in the video information of the visual field through the trained convolutional neural network model, and determine the obstacle in the image; The coordinate determination unit is used to frame the obstacles in each frame of images through a rectangular frame to obtain the coordinates of the obstacles in the image. 10. The recognition system of human motion intention according to claim 6, characterized in that, further comprising: The interpolation module is used for synchronizing the motion coordinates, the attention point coordinates and the obstacle coordinates into signals with equal intervals by using a spline interpolation method.

Images