CN112315456B - Human body action prediction method based on jump attention mechanism - Google Patents
Human body action prediction method based on jump attention mechanism Download PDFInfo
- Publication number
- CN112315456B CN112315456B CN202011067849.2A CN202011067849A CN112315456B CN 112315456 B CN112315456 B CN 112315456B CN 202011067849 A CN202011067849 A CN 202011067849A CN 112315456 B CN112315456 B CN 112315456B
- Authority
- CN
- China
- Prior art keywords
- human body
- human
- skeleton point
- decoder
- features
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 241000282414 Homo sapiens Species 0.000 title claims abstract description 88
- 230000009471 action Effects 0.000 title claims abstract description 33
- 238000000034 method Methods 0.000 title claims abstract description 21
- 230000007246 mechanism Effects 0.000 title claims abstract description 18
- 230000007774 longterm Effects 0.000 claims abstract description 24
- 239000013598 vector Substances 0.000 claims abstract description 23
- 230000033001 locomotion Effects 0.000 claims abstract description 18
- 230000004931 aggregating effect Effects 0.000 claims abstract description 5
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 4
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 2
- 210000000988 bone and bone Anatomy 0.000 description 8
- 230000000694 effects Effects 0.000 description 7
- 230000004913 activation Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000576 supplementary effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
- A61B5/1118—Determining activity level
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
- A61B5/1121—Determining geometric values, e.g. centre of rotation or angular range of movement
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7271—Specific aspects of physiological measurement analysis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7271—Specific aspects of physiological measurement analysis
- A61B5/7275—Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Public Health (AREA)
- Surgery (AREA)
- Veterinary Medicine (AREA)
- Physiology (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- Artificial Intelligence (AREA)
- Psychiatry (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Signal Processing (AREA)
- Dentistry (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Geometry (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a human body action prediction method based on a jump connection attention mechanism, which comprises the following steps: inputting the normalized human skeleton point coordinates of each frame into an encoder consisting of a multilayer self-updating convolution gating circulation network, and extracting the layering characteristics of the human skeleton point sequence; aggregating the layered features extracted at the last time step of the encoder to obtain a long-term semantic vector of the human body action; taking hidden variables of each layer of the last time step of the encoder as initial hidden variables of the decoder, and calculating the weight of each layer of characteristics of the decoder according to human body action long-term semantic vectors of the encoder by using a jump attention mechanism; connecting the adjusted hierarchical features and the human motion long-term semantic vector into new features, generating the variable quantity of a human skeleton point frame to be predicted and the previous frame through a convolutional neural network, adding the variable quantity to the input skeleton points, and then performing normalized data reduction to obtain the predicted value of the human motion skeleton points. The method can effectively predict the human body action and has high accuracy.
Description
Technical Field
The invention relates to a human body motion prediction technology, in particular to a human body motion prediction method based on a jump-over attention mechanism.
Background
The human body action prediction is to automatically predict the future action sequence according to the observed human body action sequence, and can be applied to most human-computer interaction systems. The human body action prediction has wide application scenes: the system has supplementary effect on industrial automation and automatic driving technology and has important effect on interaction of intelligent robots and human beings.
Human motion prediction has attracted more and more researchers' attention, with the main challenge that early human motion prediction tasks rarely target human behavior; the long-term human body action prediction result tends to be constant action; the prediction effect is poor for aperiodic actions.
Disclosure of Invention
The invention aims to provide a human body motion prediction method based on a jump connection attention mechanism, which has a good long-term prediction effect on periodic and aperiodic human body motions.
The technical scheme for realizing the purpose of the invention is as follows: a human body action prediction method based on a jump-over attention mechanism comprises the following steps:
step 1, inputting a section of human skeleton point coordinate sequence, and carrying out normalization processing on the human skeleton point coordinate sequence to obtain a processed section of human skeleton point coordinate sequence;
step 2, inputting the human skeleton point coordinates of each frame into an encoder consisting of a multilayer self-updating convolution gating circulation network, and extracting the layering characteristics of the human skeleton point sequence;
step 3, aggregating the hierarchical features extracted by the gated loop network of the last time step to obtain a long-term semantic vector of the human body action;
step 4, inputting the layered features extracted by the encoder and the previous frame of human skeleton points into a decoder to obtain layered features;
step 5, calculating the weight of each layer of features of the decoder according to the human body action long-term semantic vector of the encoder by using a jump attention mechanism so as to adjust the importance of the layered features;
step 6, connecting the adjusted hierarchical features and the human body action long-term semantic vectors into new features, and generating the variation of the human body skeleton point frame to be predicted and the previous frame by the new features through a convolutional neural network;
and 7, adding the variable quantity of the skeleton points and the input frame of the decoder, and then restoring the normalized data to obtain the final predicted value of the human action skeleton points.
Compared with the prior art, the invention has the remarkable advantages that; (1) the method is based on the jump attention mechanism, can effectively predict the human body action, and has high prediction accuracy; (2) the method has good long-term prediction effect on periodic and aperiodic human body actions.
Drawings
Fig. 1 is a flowchart of a human body motion prediction method based on a jump attention mechanism according to the present invention.
Fig. 2 is an effect diagram of a human body motion prediction method based on a jump attention mechanism.
Detailed Description
The invention is described in further detail below with reference to the accompanying drawings:
as shown in FIG. 1, a human body motion prediction method based on a jump attention mechanism comprises four processes of extracting human body skeleton point sequence hierarchical features, extracting long-term semantic vectors, calculating weights of decoder hierarchical features and generating skeleton point variable quantities.
Extracting human skeleton point sequence layering characteristics: inputting the normalized human skeleton point coordinates of each frame into an encoder consisting of a multilayer self-updating convolution gating circulation network, and extracting the hierarchical characteristics of the human skeleton point sequence;
extracting a long-term semantic vector: aggregating the layered features extracted at the last time step of the encoder to obtain a long-term semantic vector of the human body action;
calculating weights of decoder hierarchical features: taking hidden variables of each layer of the last time step of the encoder as initial hidden variables of the decoder, and calculating the weight of each layer of features of the decoder according to human body action long-term semantic vectors of the encoder by using a jump attention mechanism so as to adjust the importance of the hierarchical features;
generating the bone point variation: and connecting the adjusted hierarchical features and the human motion long-term semantic vector into new features, generating the variation of a human skeleton point frame to be predicted and a previous frame by the new features through a convolutional neural network, adding the variation of the skeleton point to the input skeleton point of a decoder, and restoring normalized data to finally obtain the predicted value of the human motion skeleton point. The method can effectively predict the human body action and has high prediction accuracy.
The steps of the above method will be described in detail below.
The method for extracting the human skeleton point sequence layering characteristics comprises the following steps:
step 1), inputting a section of human body skeleton point coordinate sequence, and carrying out normalization processing on the human body skeleton point coordinate sequence to obtain a processed section of human body skeleton point coordinate sequence X ═ X1,…,xs,…,xS]Which describes the activity of a person, of which there is xs∈RNAnd the skeleton key point coordinates of the human body at the time step s are shown, and N is the number of joints of the human body.
Step 2), the sequence X obtained in step 1) ═ X1,…,xs,…,xs]Is input toIn an encoder composed of a multi-layer Self-updating convolution gating circulation network (Self-updating ConvGRU), the calculation formula of one layer of the multi-layer Self-updating convolution gating circulation network (Self-updating ConvGRU) at the time step t is as follows:
updating the door ztCalculated from the formula:
zt＝σ(Wzh*ht-1+Wzx*xt+bz),
reset gate rtCalculated from the formula:
rt＝σ(Wrh*ht-1+Wrx*xt+br),
candidate hidden variables
Calculated from the formula:
self-updating door hhtCalculated from the formula:
then candidate hidden variables
Is updated again:
finally, the current hidden variable h is obtainedt：
Wherein h ist-1Is the last time step tHidden variable of-1, xtIs a hidden variable of the previous layer at time step t, σ () is a sigmoid activation function, tanh () is a tanh activation function, W*Is a learnable transformation parameter, b*Is a bias term, is a convolution operation,
are multiplied by element.
Step 3) obtaining hidden variables of each layer of the input sequence at the last time step of the encoder through the step 2)
Where K denotes the number of layers of the encoder, these hidden variables are the hierarchical characteristics of the input sequence.
The extraction of the long-term semantic vector comprises the following steps:
step 4) the hierarchical characteristics of the input sequence obtained in the step 3)
And (3) aggregating to form a human body action long-term semantic vector:
where g () is an aggregation function, a convolution operation is used as the aggregation function.
The method for calculating the weight of the decoder hierarchical characteristics mainly comprises the following steps:
step 5) constructing a decoder by adopting the same calculation formula in the step 2), taking hidden variables of each layer of the encoder obtained in the step 3) as initial hidden variables of the decoder, taking human skeleton points obtained by predicting at the time step t-1 as input of the decoder at the time step t, and finally obtaining hidden variables of the self-updating convolution gated cyclic network of each layer
Step 6), hidden variables from decoder by convolution operation
Extracting the characteristics of each layer at time step t
Step 7), calculating the weight of each layer of features of the decoder according to the long-term semantic vector of the human body action, and setting the features of the nth layer of the time step t
The score calculation formula is as follows:
wherein WfIs a weight matrix, bfIs a bias vector;
the score represents the importance of the features of the decoder layers, and is normalized using the Softmax function:
finally, the
Representing the adjusted characteristics of the decoder layers.
Generating the bone point variation comprises the following steps:
step 8), connecting the characteristics of the time step t adjusted by the decoder in the step 7) with the long-term semantic vector of the human action in the step 4), and generating the variation of the human skeleton points through convolution operation:
the predicted human body bone point coordinate at time step t +1 can be obtained by adding the human body bone point variation at time step t and the input human body bone point coordinate of the decoder.
Fig. 2 is a diagram of the effect of the human body motion prediction method based on the jump-joint attention mechanism, in which the first four rows in the first row are real bone points, the first four rows are predicted bone points from the fifth row, and the second row is real bone points for comparison.
Claims (3)
1. A human body action prediction method based on a jump-over attention mechanism is characterized by comprising the following steps:
step 1, inputting a section of human skeleton point coordinate sequence, and carrying out normalization processing on the human skeleton point coordinate sequence to obtain a processed section of human skeleton point coordinate sequence;
step 2, inputting the human skeleton point coordinates of each frame into an encoder consisting of a multilayer self-updating convolution gating circulation network, and extracting the layering characteristics of the human skeleton point sequence;
step 3, aggregating the hierarchical features extracted by the gated loop network of the last time step to obtain a long-term semantic vector of the human body action;
step 4, inputting the layered features extracted by the encoder and the previous frame of human skeleton points into a decoder to obtain layered features;
step 5, calculating the weight of each layer of features of the decoder according to the human body action long-term semantic vector of the encoder by using a jump attention mechanism so as to adjust the importance of the layered features;
step 6, connecting the adjusted hierarchical features and the human body action long-term semantic vectors into new features, and generating the variation of the human body skeleton point frame to be predicted and the previous frame by the new features through a convolutional neural network;
step 7, adding the variable quantity of the skeleton points and the input frame of the decoder, and then restoring the normalized data to obtain a final predicted value of the human body action skeleton points;
the step 7 specifically comprises the following steps:
the variation of the human skeleton point of the last time step t
And the input human skeleton point coordinate y of the last time step t decodertAdding to obtain the human skeleton point coordinate y of the predicted time step t +1t+1：
Finally, the predicted human skeleton point y of the time step t +1t+1And restoring the normalized data to finally obtain the predicted value of the human body action skeletal point at the time step t + 1.
2. The human motion prediction method based on the jump attention mechanism according to claim 1, wherein the step 5 specifically comprises the following steps:
step 501, calculating the weight of each layer of features of the decoder according to the long-term semantic vector of human body motion, and setting the features of the nth layer of a time step t
The score calculation formula is as follows:
wherein WfIs a weight matrix, bfIs a bias vector, C represents a long-term semantic vector of human body motion;
step 502, the score represents the importance of each layer feature of the decoder, and the score is normalized by using a Softmax function
：
Finally, the
Representing the adjusted characteristics of the decoder layers.
3. The human motion prediction method based on the jump attention mechanism according to claim 2, wherein the step 6 specifically comprises the following steps:
connecting the characteristics of the time step t after being adjusted by the decoder with the long-term semantic vector of the human body action, and generating the variable quantity of the human body skeleton point through convolution operation:
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011067849.2A CN112315456B (en) | 2020-10-07 | 2020-10-07 | Human body action prediction method based on jump attention mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011067849.2A CN112315456B (en) | 2020-10-07 | 2020-10-07 | Human body action prediction method based on jump attention mechanism |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112315456A CN112315456A (en) | 2021-02-05 |
| CN112315456B true CN112315456B (en) | 2022-02-11 |
Family
ID=74314478
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011067849.2A Expired - Fee Related CN112315456B (en) | 2020-10-07 | 2020-10-07 | Human body action prediction method based on jump attention mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112315456B (en) |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012008868A (en) * | 2010-06-25 | 2012-01-12 | Toshiba Corp | Display processing device, display processing method, and display processing program |
| AU2012285379B2 (en) * | 2011-07-20 | 2017-04-13 | Elminda Ltd. | Method and system for estimating brain concussion |
| US10405045B2 (en) * | 2015-12-14 | 2019-09-03 | Google Llc | Systems and methods for estimating user attention |
| US11089968B2 (en) * | 2017-04-19 | 2021-08-17 | Beijing Shunyuan Kaihua Technology Co., Ltd. | Systems and methods for providing user insights based on heart rate variability |
| US10970527B2 (en) * | 2017-09-01 | 2021-04-06 | Digital Dream Labs, Llc | Robot attention detection |
| CN109961444B (en) * | 2019-03-01 | 2022-12-20 | 腾讯科技（深圳）有限公司 | Image processing method and device and electronic equipment |
| CN110580456A (en) * | 2019-08-22 | 2019-12-17 | 南京理工大学 | Group Activity Recognition Method Based on Coherence Constraint Graph Long Short-Term Memory Network |
| CN111242048A (en) * | 2020-01-15 | 2020-06-05 | 南京理工大学 | Human skeleton action prediction method based on space-time cooperative attention circulation neural network |
-
2020
- 2020-10-07 CN CN202011067849.2A patent/CN112315456B/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN112315456A (en) | 2021-02-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ivanovic et al. | Generative modeling of multimodal multi-human behavior | |
| Zhu et al. | Masked contrastive representation learning for reinforcement learning | |
| US9082079B1 (en) | Proportional-integral-derivative controller effecting expansion kernels comprising a plurality of spiking neurons associated with a plurality of receptive fields | |
| Qiang et al. | Reinforcement learning model, algorithms and its application | |
| Gencay et al. | Nonlinear modelling and prediction with feedforward and recurrent networks | |
| CN113625757B (en) | Unmanned aerial vehicle group scheduling method based on reinforcement learning and attention mechanism | |
| Chan et al. | ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning | |
| Yesil et al. | Fuzzy cognitive maps learning using artificial bee colony optimization | |
| WO2014018793A1 (en) | Apparatus and methods for efficient updates in spiking neuron networks | |
| Huang et al. | Computational modeling of emotion-motivated decisions for continuous control of mobile robots | |
| CN111897224A (en) | A Multi-Agent Formation Control Method Based on Actor-Critic Reinforcement Learning and Fuzzy Logic | |
| Cao et al. | An overview of deep reinforcement learning | |
| CN113469891A (en) | Neural network architecture searching method, training method and image completion method | |
| Chen et al. | Policy gradient from demonstration and curiosity | |
| Schneider | Causal cognitive architecture 1: Integration of connectionist elements into a navigation-based framework | |
| Guo et al. | Dynamic neural network structure: A review for its theories and applications | |
| CN112315456B (en) | Human body action prediction method based on jump attention mechanism | |
| Taghian et al. | Explainability of deep reinforcement learning algorithms in robotic domains by using Layer-wise Relevance Propagation | |
| Dong et al. | LAMBO: Large AI Model Empowered Edge Intelligence | |
| Xia et al. | A conditional classification recurrent RBM for improved series mid-term forecasting | |
| Son et al. | A survey of brain inspired technologies for engineering | |
| Zhu et al. | Fast Adaptive Character Animation Synthesis Based on Greedy Algorithm | |
| CN115587615A (en) | An Intrinsic Reward Generation Method for Perception-Action Loop Decision-Making | |
| Liu et al. | Towards explainable reinforcement learning using scoring mechanism augmented agents | |
| CN115091467A (en) | Intent prediction and disambiguation method and system based on fuzzy Petri net |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220211 |