CN106778639A - A kind of exercise data search method based on the description of attitude relative space-time characteristic statisticses - Google Patents

Abstract

The invention discloses a motion data retrieval method based on the statistical description of the relative spatio-temporal features of postures, which is divided into the following steps: extracting the relative spatio-temporal features of the three-dimensional postures, and extracting the relative space between the geometric elements of points, lines and planes formed by joints in the three-dimensional postures The measurement of the position and its change is used as the characteristic representation of the posture; the statistical description of the movement is generated, and the statistical description of the relative spatiotemporal characteristics of the posture is extracted as the characteristic vector representation of the movement. The selected statistics include mean, range, variance and skewness. Statistical variables; similarity matching: use the Euclidean distance to calculate the similarity between the actions in the library and the retrieval actions, sort the final similarity from high to low, and return the candidate result actions; related feedback, through the linear support vector machine (SVM‑Support Vector Machine) establishes a classification model based on user feedback, thereby optimizing feature subsets and their weight combinations, making retrieval performance optimal.

Description

A Motion Data Retrieval Method Based on Statistical Description of Attitude Relative Spatiotemporal Features

technical field

The invention relates to a human body motion data retrieval method, which belongs to the field of computer three-dimensional animation technology and multimedia data processing, and specifically relates to a human body motion data retrieval method based on statistical description of posture relative spatiotemporal features.

Background technique

Due to the urgent needs of various applications and the widespread promotion of commercial capture equipment, there have been more and more large-scale 3D human motion libraries, such as the human motion library of Carnegie Mellon University (http://mocap.cs .cmu.edu), etc. How to obtain the human body motion data required by users from the human body motion library has become a key issue in the effective use of motion data. The instance-based retrieval technology uses features to represent the content of motion, and uses the similarity measurement between features to achieve the matching of motion with similar content. , overcomes the shortcomings of traditional text-based annotation retrieval methods, and has become a research hotspot in the field of motion capture data retrieval. Effective feature representation and corresponding similarity matching mechanism are the basic and key issues of instance-based retrieval.

Features should be able to fully and effectively represent the content of the motion. In the existing gesture feature representations, the three-dimensional coordinates of joints and their coded representations, quaternions and Euler angles and their coded representations are commonly used representations, but studies have shown that they are only suitable for numerically similar motion data retrieval, and Cannot effectively represent logically similar movements due to differences in style, duration, etc. Müller et al proposed Boolean geometric features in the literature Müller M, et al. Efficient content based retrieval of motion capture data. ACM Transactions on Graphics, 2005, 24(3): 677-685) to achieve logically similar motion retrieval, but limited The number of features and the binary Boolean state make it less capable of distinguishing subdivided actions. Pan Hong et al. In the literature Pan Hong, Xiao Jun, Wu Fei et al. 3D Human Motion Retrieval Based on Key Frames [J], Journal of Computer-Aided Design and Graphics, 2009,21(2):214-222[77] It is proposed to use the angle between the limb bones and the central bone as the pose feature, but there are many local actions that have nothing to do with the central bone. Tang et al pointed out in the document Tang J T, et al. Retrieval of logically relevant 3D humanmotions by Adaptive Feature Selection with Graded Relevance Feedback [J]. Pattern Recognition Letters, 2012,33(4):420-430 that the relative positional relationship is the extraction It is an effective representation of logical meaning and uses the distance between joint pairs as the feature representation of the pose. However, it ignores the two important geometric elements of straight lines and planes. In addition, this feature cannot directly express the content of joint rotation. Chen et al. defined the pose relative geometric feature set in the document ChenC, et al. Perceptual 3D pose distance estimation by boosting relational geometric features [J]. Computer Animation and Virtual Worlds, 2009, 20(2-3): 267-277. Evaluate the similarity of posture in perception. However, any combination of points, lines, and surfaces formed by different joints makes the number of features reach more than 560,000 dimensions. Reasonable constraints need to be imposed on specific applications to reduce the number of feature dimensions.

Attitude is the constituent unit of motion. Based on the feature representation of attitude, some researchers obtain the low-dimensional representation of attitude through dimensionality reduction and clustering methods; through matrix singular value decomposition, spherical harmonic transformation, tensor algebraic subspace decomposition and Keyframing techniques achieve a higher level re-description of motion clips. However, in dimensionality reduction, clustering, and calculation of the secondary feature extraction distance of actions, the local features of gestures that reflect the content of motion are all assigned specific weights, and the weights of these local features cannot be used in the retrieval process as the retrieval instance changes. Different dynamic adjustments lead to the fact that the above feature representations cannot effectively support the local space retrieval of motion.

To sum up, the existing feature representations of motion content cannot effectively support motion data retrieval. The present invention proposes a motion data instance retrieval method based on the statistical description of attitude relative spatiotemporal features. On the basis of this method, the method further adopts the secondary feature vector of the extracted action by statistical description. During the retrieval process, the method also adopts SVM-based correlation feedback to improve the retrieval performance. The method closer to the method of the present invention is literature (Chen Songle, etc. The example retrieval method of national dance data. Computer Aided Design and Graphics Journal, 2014, 26 (2): 198-210) and literature (Songle Chen, et al .Relevance Feedback for Human MotionRetrieval Using a Boosting Approach.Multimedia tools and applications, 2016,75(2):787-817), however, both methods directly use the attitude relative spatiotemporal features as feature representations, and use As a motion similarity calculation method, the dynamic time warping algorithm is too complex to meet the performance requirements of large-scale motion capture database retrieval.

Contents of the invention

Purpose of the invention: The technical problem to be solved by the present invention is to propose a motion data retrieval method based on the statistical description of the relative spatio-temporal features of postures in view of the deficiencies of the prior art.

Technical solution: The invention discloses a motion data retrieval method based on the statistical description of posture relative spatio-temporal features, comprising the following steps:

Step 1, 3D pose relative spatio-temporal feature extraction: the set of point, line, and surface geometric elements formed by joints in 3D pose is the smallest constituent unit of the local area corresponding to different action modes. The present invention extracts the relative spatial position between the geometric elements of points, lines and surfaces formed by the joints in the three-dimensional posture and the measurement of their changes as the feature representation of the posture, and expresses a wide range of features through the weight combination of different types of features contained in different local areas. Attitude mode. For each gesture contained in all actions in the library, the specific steps of its relative spatiotemporal feature extraction are as follows:

1) Define the three-dimensional human joint model, and select the most important joints as the three-dimensional pose representation;

2) Construct a set of geometric elements. The selected joints constitute a point set in the set of geometric elements. Any 2 points in the point set form a straight line, and any 3 points form a plane;

3) Extract the relative spatial features of each 3D pose, including joint pair distance feature, joint and bone distance feature, joint and plane distance feature, bone pair angle feature, bone and plane angle feature, plane and plane angle feature, joint rotation feature;

4) Extract relative time features of each three-dimensional attitude, including joint angular velocity and acceleration features.

Step 2, Action Statistical Description Generation: The action is formed by continuous changes of several postures. The present invention extracts the statistical description of the relative spatiotemporal characteristics of the posture as the feature vector representation of the action. The selected statistics include mean, range, variance and skewness There are 4 statistical variables in total. For each action in the library, the specific steps of action statistics description generation are as follows:

1) Load the relative spatiotemporal features of each three-dimensional pose contained in the action, and use each feature as a one-dimensional random variable, and each pose in the action corresponds to the value of the random variable;

2) Calculate the mean of each relative spatiotemporal feature;

3) Calculate the range of each relative spatio-temporal feature;

4) Calculate the variance of each relative spatiotemporal feature;

5) Calculate the skewness of each relative spatiotemporal feature;

6) The mean value, range, variance, and skewness of each relative spatiotemporal feature extraction are normalized and arranged in order and saved to form a feature vector representation of each action.

Step 3, similarity matching: use step 1 and step 2 to extract the feature description of the retrieval action instance submitted by the user, and then use the Euclidean distance to calculate the similarity between the action and the retrieval action in the database, and sort the final similarity from high to low Return the result action. The specific steps of the similarity matching process are as follows:

1) The user submits a retrieval action instance;

2) Use steps 1 and 2 to calculate the statistical description of the relative spatiotemporal characteristics of the retrieval action submitted by the user;

3) For each action in the library, based on the statistical description of relative spatio-temporal features, the Euclidean distance is used to calculate the distance from the search action submitted by the user, and the weight of each feature is the same;

4) Sort the distance between each action in the library and the retrieval action submitted by the user;

5) Return the Top-20 actions with the smallest distance to the user.

Step 4, relevant feedback: In the similarity calculation of step 3, each feature uses the same weight. In fact, there is a specific feature subset and its weight combination for each user's retrieval, which makes the retrieval performance optimal. In order to better capture the user's retrieval intention and gradually approach the optimal feature subset and its weight combination, the present invention optimizes the retrieval results through the SVM-Support Vector Machine (SVM-Support Vector Machine) correlation feedback method. The correlation feedback based on SVM is specific Proceed as follows:

1) For the result of step 3 or the previous round of relevant feedback, the user marks whether the returned instance is related to the search action submitted by him;

2) The samples marked by the user that are related to the submitted retrieval action are taken as positive samples, and the samples marked by the user that are not related to the submitted search action are taken as negative examples;

3) Select the linear kernel, use SVM to learn positive and negative samples, and obtain the SVM classification model;

4) Input each action in the library to the SVM classification model in turn, and calculate the score;

5) Sort the scores and return the Top-20 actions with the highest scores to the user;

6) If the user is satisfied with the result, the retrieval process ends, otherwise return to step 1) for the next round of feedback.

Beneficial effect

1) The present invention adopts the spatial-temporal relative feature of the gesture as the three-dimensional gesture description, and the feature description extracts the relative spatial position between the geometric elements formed by the joints in the gesture, the line, and the surface and the measurement of the change thereof as the action content representation. The set of point, line, and surface geometric elements formed by joints is the smallest constituent unit of the local area corresponding to different action modes, and the measures such as angle and distance between point, line, and surface geometric elements reflect the relationship between the smallest constituent units from different aspects. A wide range of action patterns can be expressed through the weight combination of different types of features contained in different local regions.

2) The present invention uses the statistical description of the spatial-temporal relative characteristics of three-dimensional postures as the feature representation of the action. The mean, range, variance, and skew statistical variables reflect the benchmark and change of each feature in the action, which can effectively express the content of the action . At the same time, the statistical description uniformly converts actions of different lengths in time into feature vectors of uniform length, so methods such as Euclidean distance and linear SVM can be used for similarity calculation and related feedback, which greatly improves the computational efficiency and retrieval effectiveness. .

3) The present invention uses linear SVM for correlation feedback to further improve retrieval performance. SVM itself has the advantage of good generalization ability of small sample learning, so it can overcome the poor generalization ability of general learning methods caused by the lack of sample size marked by users The problem. The use of the linear kernel function makes it possible to explain the different important programs for each feature in this retrieval according to the weight of each feature in the classification model, which improves the interpretability of the retrieval results.

Description of drawings

Fig. 1 is a flow chart of the method of the present invention.

Fig. 2 is a schematic diagram of a human joint model used in the present invention.

Fig. 3 is a schematic diagram of relative spatio-temporal features defined in the present invention.

Fig. 4 is the retrieval result of the retrieval input action "jump" obtained through similarity matching in the embodiment.

Fig. 5 is the result of searching for the retrieval action "jump" for 4 iterations through SVM related feedback in the embodiment.

detailed description

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

A motion data retrieval method based on statistical description of posture relative spatiotemporal features, comprising the following steps:

Step 1, 3D pose relative spatio-temporal feature extraction: the set of point, line, and surface geometric elements formed by joints in 3D pose is the smallest constituent unit of the local area corresponding to different action modes. The present invention extracts the relative spatial position between the geometric elements of points, lines and surfaces formed by the joints in the three-dimensional posture and the measurement of their changes as the characteristic representation of the posture. Capable of expressing a wide range of pose patterns. For each gesture contained in all actions in the library, the specific steps of its relative spatiotemporal feature extraction are as follows:

1) Define a three-dimensional human joint model. The three-dimensional human joint model adopted in the present invention is shown in FIG. 2 and includes 18 joints in total. According to the specific translation or rotation value of each joint at the sampling time, calculate the three-dimensional coordinates (x, y, z) of each joint at the sampling time;

2) Construct a set of geometric elements. The selected joints constitute a point set in the set of geometric elements. Any 2 points in the point set form a straight line, and any 3 points form a plane. The present invention reasonably constrains the two geometric elements of straight lines and planes, and only considers the straight lines and planes formed by adjacent joints to reduce the dimension of posture features. The final set of posture geometric elements contains a total of 17 straight lines formed by bones. , and 10 planes formed by adjacent joints;

3) Extract the relative spatial features of each 3D pose, including joint pair distance feature, joint and bone distance feature, joint and plane distance feature, bone pair angle feature, bone and plane angle feature, plane and plane angle feature, joint Rotate feature. The extracted 3D posture relative spatial features are shown in Figure 3, and the specific calculation process is as follows;

a) Joint pair distance feature F _{j, j, d} , the present invention uses Euclidean distance to calculate the distance between joint pairs in the geometric element set, and the three-dimensional coordinates of joints j ₁ and j ₂ in the posture are respectively (x ₁ , y ₁ ,z ₁ ), (x ₂ ,y ₂ ,z ₂ ), the formula for calculating the distance between joint pairs is:

b) The distance feature F _j,l,d between the joint and the bone. The distance from the joint to the bone is calculated by the triangle area formula. Let d ₁₂ , d ₁₃ , and d ₂₃ be the distances between joints j ₁ , j ₂ , and j ₃ respectively , p=(d ₁₂ +d ₂₃ +d ₁₃ )/2, then the distance between joint j ₁ and the line formed by joints j ₂ and j ₃ is:

F _j,l,d = 2p(pd ₁₂ )(pd ₁₃ )(pd ₂₃ )/d ₂₃ ;

c) The distance feature F _j,p,d between the joint and the plane. The distance from the joint to the plane is obtained by the dot product between the vector formed by the joint and any point on the plane and the normal vector of the plane. Let n be joints j ₂ and j ₃ , the normal vector of the plane formed by j ₄ , v is the vector formed by joints j ₁ and j ₃ , then the distance from joint j ₁ to the plane formed by j ₂ , j ₃ and j ₄ is:

F _j,p,d = n·v/||n||;

d) Bone pair angle feature F _l,l,a , the angle between bones and bones is calculated by the vector dot product formula, if joints j ₁ and j ₂ form vector v _a , joints j ₃ and j ₄ form vector v _b , the formula for calculating the included angle between bones is:

F _l,l,a = arccos(v _a ·v _b /(||v _a ||×||v _b ||));

e) The angle feature F _l,p,a between the bone and the plane, the angle between the bone and the plane is calculated by the dot product formula of the normal vector of the bone and the plane, let n be the joint j ₃ , j ₄ , j ₅ formed The normal vector of plane P, v is the vector formed by joints j ₁ and j ₂ , then the angle between the bone formed by joints j ₁ and j ₂ and P is:

F _l,p,a = arccos(n·v/(||n||×||v||));

f) The angle feature F _p,p,a between the plane and the plane, the angle between the plane and the plane is calculated by the dot product formula of the normal vector of the plane, let n ₁ be the joint j ₁ , j ₂ , j ₃ formed The normal vector of the plane P ₁ , n ₂ is the normal vector of the plane P ₂ formed by the joints j ₄ , j ₅ , j ₆ , then the angle between P ₁ and P ₂ is:

F _p,p , _a = arccos(n ₁ ·n ₂ /(||n ₁ ||×||n ₂ ||));

g) Joint rotation feature _Feuler , the above posture space features only have one-dimensional information, and cannot reflect three-dimensional adjacent joint rotation information. The present invention uses Euler angles to represent the adjacent joint rotation information.

4) Extract relative time features of each three-dimensional attitude, including joint angular velocity and acceleration features. The present invention adopts the method proposed in Kim TH, Park SI, Shin S Y. Rhythmic-motion synthesis based on motion-beat analysis [J]. Acm Transactions on Graphics, 2003,22 (3): 392-401 by Kim et al. To calculate the angular velocity and acceleration of the joint. The schematic diagram of the extracted angular velocity and acceleration features is shown in Figure 3. Assume that the rotations of joint j at time i-1 and i are represented by quaternions as q _j (i-1) and q _j (i) respectively, and the sampling interval is Δt.

a) The angular velocity of joint j at time i is:

b) After calculating the angular velocity, the acceleration can be obtained according to the change of the angular velocity, namely:

Step 2, Action Statistical Description Generation: The action is formed by continuous changes of several postures. The present invention extracts the statistical description of the relative spatiotemporal characteristics of the posture as the feature vector representation of the action. The selected statistics include mean, range, variance and skewness There are 4 statistical variables in total. For each action in the library, the specific steps of action statistics description generation are as follows:

1) Load the relative spatio-temporal features of each 3D pose included in the action, and use each feature as a 1D random variable, each pose in the action corresponds to the value of the random variable, n is the number of poses included in the action , the n gestures contained in the action have values of x ₁ , x ₂ ,...,x _n with respect to the relative spatio-temporal features of a gesture;

2) Calculate the mean of each relative spatiotemporal feature:

3) Calculate the range of each relative spatiotemporal feature:

r=max(x ₁ , x ₂ , . . . , x _n )-min(x ₁ , x ₂ , . . . , x _n );

4) Calculate the variance of each relative spatiotemporal feature:

5) Calculate the skewness of each relative spatiotemporal feature, which is a measure of the shape of the data distribution, and the calculation method is:

6) Normalize the mean value, range, variance, and skewness of each relative spatiotemporal feature extraction, and then arrange and save them in order to form a feature vector representation of each action.

Step 3, similarity matching: use step 1 and step 2 to extract the feature description of the retrieval action instance submitted by the user, and then use the Euclidean distance to calculate the similarity between the action and the retrieval action in the database, and sort the final similarity from high to low Return the result action. The specific steps of the similarity matching process are as follows:

1) The user submits a retrieval action instance;

2) Use steps 1 and 2 to calculate the statistical description of the relative spatiotemporal characteristics of the retrieval action submitted by the user;

3) For each action in the library, based on the statistical description of the relative spatio-temporal features, the Euclidean distance is used to calculate the distance from the search action submitted by the user, and each feature has the same weight. Suppose the user’s retrieval action is i, the candidate action in the library is j, N _f is the total number of 3D pose relative spatiotemporal features, f _i,k,l and f _j,k,l are action i and action j’s relative spatiotemporal features The statistics of the feature f describe the value of k, then the specific calculation method for the distance between the retrieval action i and the candidate action j in the library is:

4) Sort the distance between each action in the library and the retrieval action submitted by the user;

5) Return the Top-20 actions with the smallest distance to the user.

Step 4, relevant feedback: In the similarity calculation of step 3, each feature uses the same weight. In fact, there is a specific feature subset and its weight combination for each user's retrieval, which makes the retrieval performance optimal. In order to better capture the user's retrieval intention and gradually approach the optimal feature subset and its weight combination, the present invention optimizes the retrieval results through the SVM-Support Vector Machine (SVM-Support Vector Machine) correlation feedback method. The correlation feedback based on SVM is specific Proceed as follows:

1) For the result of step 3 or the previous round of relevant feedback, the user marks whether the returned instance is related to the search action submitted by him;

2) The samples marked by the user that are related to the submitted retrieval action are taken as positive samples, and the samples marked by the user that are not related to the submitted search action are taken as negative examples;

3) Select the linear kernel, use SVM to learn positive and negative samples, and obtain the SVM classification model;

4) Input each action in the library to the SVM classification model in turn, and calculate the score;

5) Sort the scores and return the Top-20 actions with the highest scores to the user;

6) If the user is satisfied with the result, the retrieval process ends, otherwise return to step 1) for the next round of feedback.

Figure 4 and Figure 5 show the effect of the motion retrieval system implemented in this scheme on the "jump" action. Figure 4 shows the result of using similarity matching. Since each feature uses the same weight, the weight combination of features It is not optimized, so out of 9 search result actions displayed on the home page, only 4 are relevant. Figure 5 shows the results after 4 iterations of linear SVM correlation feedback. By optimizing the combination of feature weights through the online learning classification model, the retrieval performance has been significantly improved. All 9 retrieval result actions displayed on the home page are related actions.

The present invention provides a motion data retrieval method based on the statistical description of attitude relative to spatio-temporal features. There are many methods and approaches to specifically realize the technical solution. The above description is only a preferred embodiment of the present invention. Those of ordinary skill may make some improvements and modifications without departing from the principle of the present invention, and these improvements and modifications shall also be regarded as the protection scope of the present invention. All components that are not specified in this embodiment can be realized by existing technologies.

Claims (5)

1. A motion data retrieval method based on attitude relative spatio-temporal feature statistical description, is characterized in that, comprises the following steps: Step 1, 3D posture relative spatio-temporal feature extraction: extract the relative spatial position and the measurement of the change between the point, line and surface geometric elements formed by the joints in the 3D posture as the feature representation of the posture, and use different types of features contained in different local regions The combination of weights to express a wide range of pose patterns; Step 2, action statistical description generation: extract the statistical description of the relative spatio-temporal characteristics of the posture as the feature vector representation of the action, the selected statistics include: value, range, variance and skewness, a total of 4 statistical variables; Step 3, similarity matching: use step 1 and step 2 to extract the feature description of the retrieval action instance submitted by the user, and then use the Euclidean distance to calculate the similarity between the action and the retrieval action in the database, and sort the final similarity from high to low Return the result action; Step 4, relevant feedback: In the similarity calculation of step 3, each feature uses the same weight. In fact, there is a specific feature subset and its weight combination for each retrieval of the user. Correlation through the support vector machine The feedback method is used to optimize the retrieval results to better capture the user's retrieval intention and gradually approach the optimal feature subset and its weight combination. 2. The method according to claim 1, wherein said step 1 comprises: 1-1) Define a three-dimensional human joint model, and select the most important joints as three-dimensional pose representations; 1-2) Construct a set of geometric elements, the selected joints constitute a point set in the set of geometric elements, any 2 points in the point set form a straight line, and any 3 points form a plane; 1-3) Extract the relative space features of each three-dimensional posture, including joint pair distance features, joint and bone distance features, joint and plane distance features, bone pair angle features, bone and plane angle features, plane and plane angle features , joint rotation characteristics; 1-4) Extract relative time features of each three-dimensional attitude, including joint angular velocity and acceleration features. 3. The method according to claim 1, wherein said step 2 comprises: 2-1) Load the relative spatio-temporal features of each three-dimensional pose contained in the action, and use each feature as a one-dimensional random variable, and each pose in the action corresponds to the value of the random variable; 2-2) Calculate the mean value of each relative spatiotemporal feature; 2-3) Calculate the range of each relative spatio-temporal feature; 2-4) Calculate the variance of each relative spatiotemporal feature; 2-5) Calculate the skewness of each relative spatiotemporal feature; 2-6) The mean value, range, variance, and skewness of each relative spatiotemporal feature extraction are normalized and then arranged and saved in order to form a feature vector representation of each action. 4. The method according to claim 1, wherein said step 3 comprises: 3-1) The user submits a retrieval action instance; 3-2) Step 1 and Step 2 are used to calculate the statistical description of the relative spatio-temporal characteristics of the retrieval action submitted by the user; 3-3) For each action in the library, based on the statistical description of the relative spatio-temporal features, the Euclidean distance is used to calculate the distance from the search action submitted by the user, and the weight of each feature is the same; 3-4) Sorting the distance between each action in the library and the retrieval action submitted by the user; 3-5) Return the Top-20 actions with the smallest distance to the user. 5. The method according to claim 1, wherein said step 4 comprises: 4-1) For the result of step 3 or the previous round of related feedback, the user marks whether the returned instance is related to the search action submitted by him; 4-2) The samples marked by the user that are related to the search action submitted by the user are regarded as positive samples, and the samples marked by the user that are not related to the search action submitted by the user are regarded as negative examples; 4-3) Select a linear kernel, use SVM to learn positive and negative samples, and obtain an SVM classification model; 4-4) input each action in the library to the SVM classification model in turn, and calculate the score; 4-5) Sort the scores, and return the Top-20 actions with the highest scores to the user; 4-6) If the user is satisfied with the result, the retrieval process ends, otherwise return to step 4-1) for the next round of feedback.