CN100543775C - Method of 3D Human Motion Tracking Based on Multi-camera - Google Patents

Abstract

The method that a kind of 3 d human motion based on many orders camera of technical field of computer vision is followed the tracks of, step is: the first, handle voxel data at the number of voxels strong point of extracting human body surface, obtains outermost point; The second, before being followed the tracks of, human motion makes up the three-dimensional human skeleton model of a standard carrying out skeleton extract, be used in the middle of the process of following the tracks of, it being carried out dynamic stance adjustment, to realize it and original three-dimensional image match; The 3rd, make up the human skeleton observation procedure, whether be the maximum skeleton pattern attitude of adjusting according to the tissue points number in every section bone sleeve; The 4th, the implementation method of structure Optimum Matching skeleton judges around the anglec of rotation of its father node whether current human skeleton attitude has reached tracer request by locus and every section bone of changing root node.The present invention obtains the three-dimensional framework of human body rapidly on the basis of existing human body voxel data, thereby human motion is followed the tracks of.

Description

Method of 3D Human Motion Tracking Based on Multi-camera

technical field

The invention relates to a method in the technical field of telecommunications, in particular to a method for tracking three-dimensional human body motion based on a multi-eye camera.

Background technique

Three-dimensional human motion tracking is a hot and difficult point in the field of computer vision. Extracting human skeleton models from video data is a very effective monitoring and tracking method. The main application fields of human skeleton obtained from 3D voxel data are: surveillance system, virtual reality, advanced user interface, intelligent environment, entertainment, motion analysis, medicine, education, etc. The English word for three-dimensional voxel is voxel, which is the abbreviation of the two words Volume Element. Voxel is called "voxel", which is equivalent to "pixel" in two-dimensional images - "pixel", which is the basic small element in three-dimensional space. The square contains information such as the X, Y, and Z coordinates of each point. It is a very novel method to obtain three-dimensional human voxel data through multi-eye cameras for pose estimation and human motion tracking. However, only the voxel data of 3D human body is not enough to solve many practical problems. The voxel data obtained by the multi-eye camera is directly reconstructed and tracked. Due to the difference in the number of cameras, the results are uneven, the data volume is generally large, the calculation speed is slow, and it is not easy to transmit data in real time and publish it on the Internet. Less than the effect of tracking and monitoring. Human motion tracking includes both two-dimensional human motion tracking and three-dimensional human motion tracking. 3D human motion tracking is much more complex than 2D. It is very common to analyze human motion through voxel data, but it is rarely involved in tracking human motion by obtaining accurate human skeleton through voxel data. The significance of this work is rather special, as the obtained tracking results can be used in the monitoring system, and can also be used as intermediate results to identify human body poses and actions for further research. In general, accurate tracking of human motion is of great significance and is a difficult problem in the entire field of computer vision.

After searching the existing technical literature, it was found that Caillette, F. et al. in 2004 in "Third IEEE and ACM International Symposium on Mixed and Augmented Reality" (IEEE and ACM Third International Symposium on Mixed and Extended Reality) The published paper "Real-time markerless human body tracking using colored voxels and 3D blobs", (using colored voxels and 3D modules to track unmarked human bodies in real time) uses colored voxel information to track the human body. The method needs to have a relatively good color contrast between the tracked target and the scene, which greatly limits its application in engineering.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies in the prior art, to provide a method for three-dimensional human body motion tracking based on a multi-eye camera, which has good stability, good speed, high precision, small output file, easy to save and Transmission characteristics. The present invention accurately judges and predicts the posture and action of the human body by tracking the movement of the human body, so that the computer can analyze and judge by itself. In a monitoring system, this information can be transmitted to the monitoring terminal in real time, so that Detect abnormalities in real time and take corresponding measures in time.

The present invention is achieved through the following technical solutions, comprising the following steps:

First, extract the voxel data points on the surface of the human body, process the voxel data, and obtain the outermost points;

Second, construct a standard 3D human skeleton model before extracting the skeleton to track the human body movement, which is used to dynamically adjust its posture during the tracking process to achieve its fitting with the original 3D image;

Third, construct a human skeleton extraction method, and adjust the posture of the skeleton model according to whether the number of voxel points in each bone sleeve in the three-dimensional human skeleton model is the maximum;

Fourth, construct an optimal matching skeleton implementation method, by changing the spatial position of the root node in the 3D human skeleton model and the rotation angle of each bone around its parent node to determine whether the current human skeleton posture has met the tracking requirements.

的点全部算作是该点的邻域点，共计26个。在判断过程当中判断该点的邻域点是否有背景点，若有则为最外层点。The extraction of voxel data points on the surface of the human body, and processing the voxel data to obtain the outermost point refers to: after using 16 cameras to obtain the three-dimensional human body voxel data, in order to reduce the data and improve the processing speed, the The outermost voxel of the human body is searched out. With 3D voxel data, a 3D image can be obtained by displaying all the voxels on a computer. In order to obtain the outermost point of the three-dimensional image, the present invention designs a 3*3*3 image template. Wherein 3×3×3 refers to a total of 27 points forming a cube template in the six directions of front, back, left, right, up and down of each voxel (including itself) in three-dimensional space. Unlike 2D images, this template is a 26-connected domain. For any voxel, the Euclidean distance between up, down, left, right, front, back and it is less than or equal to

The points of are all counted as the neighborhood points of this point, a total of 26 points. During the judging process, it is judged whether there is a background point in the neighborhood of the point, and if so, it is the outermost point.

The construction of a standard three-dimensional human skeleton model refers to designing a standard three-dimensional human skeleton model before performing skeleton extraction to track human motion. In this skeleton model, several nodes are defined, and a root node is defined to control the spatial rotation and spatial translation of the entire skeleton. Each node defines a parent node, and each skeleton rotates and translates around its parent node in the local coordinate system. Finally, the change of the spatial position of each node relative to the root node can be obtained, so that each The coordinates of the nodes in 3D space, which in turn determine the skeleton pose for each frame. For each bone, it has several parameters to determine, such as ID number, parent node, length and so on. Each bone defines a unique ID number, through which the current node can be connected to the parent node; the length of each bone is different, which can be obtained through setting or machine learning.

The method for constructing a human skeleton extraction refers to judging the effect of human body motion tracking according to the matching degree of the human body model and the observed features of the skeleton. Through the tracking results of the previous frame and the spatial position of the root node of this frame and the rotation angle parameters of each bone around its parent node, the human body motion of the next frame is tracked. The parameters of the first frame are obtained through human-computer interaction or machine learning. Due to the gradual change of human body motion, the parameter changes of the two frames before and after are usually within a certain range, and this range can be artificially specified according to common sense. A simple but effective method is adopted: put a sleeve on each bone, calculate the number of voxel points in the sleeve, and stop as long as it reaches the maximum and the obtained skeleton is within the surface point of the human body According to the calculation, this skeleton pose is considered to be the optimal skeleton pose, and the purpose of tracking the human body movement through the skeleton pose is achieved.

The implementation method of constructing the optimal matching skeleton refers to: in order to achieve the best matching between the human skeleton and the three-dimensional voxels of each frame, determine the exact spatial position of each node of the skeleton in each frame, so that the translation of the entire skeleton must be controlled and the three-dimensional coordinates of the rotated root node. After the first frame of 3D human skeleton is obtained, the skeleton pose of the next frame is adjusted based on the skeleton of the previous frame. The parameters for changing the skeleton pose of each frame are mainly to rotate each bone around the local coordinate system where its parent node is located around the X, Y, and Z coordinate axes. The angle of the parent node is iterated until the obtained human skeleton is inside the human body and can represent the posture of the human body. Each bone has a local optimal value, and when all the bones reach the local optimal value, the whole skeleton posture reaches the optimal value, the tracking of this frame ends, and the tracking of the next frame starts until the end of the last frame.

Compared with the prior art, the present invention is simple and effective, and its key lies in rapidly obtaining the three-dimensional skeleton of the human body on the basis of the existing human body voxel data, so that the movement posture of the human body can be clearly judged and the human body movement can be adjusted accordingly. track. The human body movement tracking by using the present invention can not only be used as the result, but also can be used as the intermediate result of the pattern recognition in the next step. Since the data volume of each frame of human skeleton file is small (less than 1K), and the size of each frame of three-dimensional image file is large (more than 5M), the obtained human skeleton file can greatly save storage space, save transmission time and transmission time in the network Cost, especially suitable for monitoring systems, but also can be applied to computer animation, games, virtual reality and other fields.

Description of drawings

Fig. 1 is a schematic diagram of the spatial position of the camera and the object according to the embodiment of the present invention

Fig. 2 is a schematic diagram of a 3×3×3 image template according to an embodiment of the present invention

Fig. 3 is a group diagram of three-dimensional human body voxel images according to the embodiment of the present invention, wherein: from left to right, each image name is the 15th frame, the 45th frame, the 75th frame, the 105th frame, the 135th frame, and the 165th frame .

Fig. 4 is the standard skeleton model diagram of the embodiment of the present invention

Fig. 5 is the schematic diagram of the skeleton sleeve of the embodiment of the present invention

Fig. 6 is a group diagram of the results of the embodiment of the present invention (only skeleton)

Among them: from left to right, the names of each image are the 15th frame, the 45th frame, the 75th frame, the 105th frame, the 135th frame, and the 165th frame.

Fig. 7 Example result group diagram (skeleton and voxel)

Among them: from left to right, the names of each image are the 15th frame, the 45th frame, the 75th frame, the 105th frame, the 135th frame, and the 165th frame.

Detailed ways

The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings: the present embodiment is implemented on the premise of the technical solution of the present invention, and detailed implementation methods and processes are provided, but the protection scope of the present invention is not limited to the following implementations example.

1. Extract human body surface voxels. As shown in Figure 1, 16 cameras are arranged in a scene. In the figure, the cameras are divided into upper and lower layers. Two-dimensional images of the human body are obtained through each camera and a three-dimensional image is constructed from these images. Each primitive that composes a three-dimensional image is voxel. In order to reduce the computational expense of the computer, increase the processing speed of the whole method, and reduce the processing time, the data points of each frame are processed using the 3×3×3 template as shown in Figure 2. The solid circles in the figure represent the voxels to be detected currently , the hollow circle represents a total of 26 adjacent voxels around it. The entire video has a total of 191 frames, and the image composed of the outermost voxels shown in Figure 3 is an image taken every 30 frames from the 15th frame. In order to display the implementation results more clearly, this embodiment adopts a method of taking every ten points to display the original data. Specific steps are as follows:

(1) First, a complete scan is performed on each frame of 3D voxel data, and the maximum and minimum values of the coordinates of the points in the X, Y, and Z directions are respectively found.

(2) According to the calculated maximum value and minimum value, the maximum value in each direction is subtracted to obtain M, N, P respectively, and a three-dimensional grid space of M×N×P dimensions is established, and the voxel is set to 1 , otherwise set to 0.

(3) Use a 3×3×3 template to process each frame of voxels, and change the value of the non-outermost point from 1 to 0, so as to obtain the target point.

2. Construct a standard human skeleton model

(1) Determine the basic information of the human skeleton. As shown in Fig. 4, it is the standard skeleton of the embodiment of the present invention, and it is made up of several bones, and the length of each bone determines different lengths respectively according to different people, marks the ID number of every section skeleton, determines the distance between the nodes. mutual relationship.

(2) In this standard skeleton model, define several nodes. The root node controls the space rotation and translation of the entire skeleton in the world coordinate system. Each skeleton is rotated and translated around its parent node in the local coordinate system, and finally the change of the spatial position of each node relative to the root node can be obtained. For the transformation matrix that rotates around an axis that does not coincide with the coordinate axis, it is obtained by compounding translation and coordinate axis rotation. For general 3D rotations: translate the skeleton to coincide with a coordinate axis parallel to the skeleton; complete the specified rotation for that axis; translate the object to move the skeleton back to its original position. Each bone can be translated and rotated around the parent node to complete the adjustment of the posture, so as to achieve the purpose of tracking.

3. Construction of human skeleton extraction method

In the process of matching the human body model and skeleton observation features, the human body motion of the next frame is tracked by using the tracking results of the previous frame and using the spatial position of the root node of this frame and the rotation angle of each bone around its parent node. The parameters of the first frame are obtained through human-computer interaction or machine learning. Through the probabilistic genetic algorithm, after setting the two parameters of the population size and the number of iterations, the skeleton of the previous frame is used as a reference for the voxel data of the next frame to perform calculations, and the skeleton of the next frame can be quickly obtained, so as to perform human motion track. As shown in Figure 5, put a sleeve on each bone, assuming that the human skeleton has N skeletons in total, then put N sleeves on the skeleton, and calculate the number of voxel points in each skeleton sleeve, The defined skeleton observation function is as follows:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; ,</span>$ ${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}& {\mathrm{<span\; class="notranslate">\; voxels</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\\ \mathrm{<span\; class="notranslate">\; 0</span>}& {\mathrm{<span\; class="notranslate">\; voxels</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&NotElement;</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\end{array}\right.$

K represents the number of defined bones, N represents the number of points in the Voxel data,

s=[s ₁ , s ₂ ,..., s _k ], s represents the whole skeleton, s _i represents the bone,

U(s _i ) represents the space inside the skeleton sleeve.

During the optimization process, the tracking effect is considered to be the best when the number of three-dimensional points in the cylinder is the largest. The tracking of the t-th frame is equivalent to finding the skeleton parameters with the best skeleton effect in the feasible region of all skeleton parameters. Since the parameter s has a high dimension and the terrain surface of the skeleton observation function is usually multi-peaked and contains many local extremums, traditional It is difficult for the optimization algorithm to find the global optimal point, so the probabilistic genetic algorithm is used to optimize the skeleton observation function to achieve the purpose of tracking.

4. The realization method of constructing the optimal matching skeleton

The exact spatial position of each node of each frame skeleton is determined, and the spatial three-dimensional coordinates of the root node can be obtained manually or by machine learning. Using the probabilistic genetic algorithm and combining the three-dimensional human motion analysis, the rotation angles of the three coordinate axes of the coordinate system where the parent node of each bone is located are compound-bit coded, and the steps are as follows:

1) Analyze the human skeleton and determine the number of individuals to be coded in combination with the limit of the actual joint motion of the person, so that the processing speed can be improved without affecting the result and time can be saved;

2) Composite bit coding is performed on the parameters (angles) that must be used by each bone;

3) Determine the population size and the number of iterations. The higher the population size, the more iterations, the more the human skeleton pose matches the original 3D image;

4) Generate the initial population (the set of rotation angles of each bone around the corresponding parent node) and observe it, and use the skeleton observation function to observe the number of voxels in the sleeve;

5) If the number increases first and then becomes stable during the iterative process, then go to step 6); if it keeps growing, a new population needs to be generated, and then return to the previous step;

6) The iteration ends.

The results of the above embodiment are shown in Figure 6 and Figure 7: each frame of image from left to right in Figure 6 corresponds to each frame of image from left to right in Figure 3, for the obtained tracking skeleton, each frame from left to right in Figure 7 The image also corresponds to each frame from left to right in Figure 3, but it also shows the resulting tracked skeleton. During the implementation process, the basic movements of a person in a three-dimensional space are captured, including basic postures such as normal walking, reaching out, and turning. Each image frame contains about 45000 3D points. The implementation results show that not only the skeleton information can be obtained well without the voxel color information, but also the additional calculation overhead caused by using this is saved. It takes about 1 minute to obtain the skeleton data of each frame of voxel data, which basically meets the real-time requirements and is much better than other methods. It can be seen that all the skeletons obtained by the present invention are within the human body and can 100% reflect the movement posture of the human body at the moment, and can play a key role in the next step of action recognition and data transmission.

Claims (9)

1, a kind of method of following the tracks of based on the 3 d human motion of many orders camera is characterized in that, may further comprise the steps:

The first, handle voxel data at the number of voxels strong point of extracting human body surface, obtains outermost point;

The second, before being followed the tracks of, human motion makes up the three-dimensional human skeleton model of a standard carrying out skeleton extract, be used in the middle of the process of following the tracks of, it being carried out dynamic stance adjustment, to realize the match of it and original three-dimensional image;

The 3rd, make up the human skeleton abstracting method, whether be the maximum skeleton pattern attitude of adjusting according to the tissue points number in every section bone sleeve in the three-dimensional human skeleton model;

The 4th, the implementation method of structure Optimum Matching skeleton judges around the anglec of rotation of its father node whether current human skeleton attitude has reached tracer request by locus and every section bone of changing root node in the three-dimensional human skeleton model.

2, method of following the tracks of according to claim 1 based on the 3 d human motion of many orders camera, it is characterized in that, the number of voxels strong point of described extraction human body surface, voxel data is handled, obtain outermost point, be meant: obtain the 3 D human body voxel data and obtain 3-D view with the computing machine demonstration by 16 video cameras, adopt 3 * 3 * 3 image template to obtain the outermost point of 3-D view then, wherein 3 * 3 * 3 be meant each voxel in three dimensions, comprise itself before, after, a left side, right, on, following six direction is formed 27 points that amount to of cube template, this template is 26 connected domains, for any one voxel, to last, down, a left side, right, before, back and its Euclidean distance are less than or equal to

Point all to can be regarded as be this neighborhood of a point point, amount to 26, judge in the middle of deterministic process whether this neighborhood of a point point has powerful connections a little, then be the outermost layer point as if having.

3, method of following the tracks of according to claim 2 based on the 3 d human motion of many orders camera, it is characterized in that, the three-dimensional human skeleton model of a standard of described structure, be meant: in the middle of the three-dimensional human skeleton model, definition plurality of nodes and a root node, control the space rotation and the spatial translation of whole skeleton, father node of each node definition, every section skeleton is all done rotation and translation around its father node in local coordinate system, draw of the change of each node at last with respect to the locus of root node, thereby determine each node at three-dimensional coordinate, and then determine the skeleton posture of every frame.

4, method of following the tracks of according to claim 3 based on the 3 d human motion of many orders camera, it is characterized in that, described every section bone, it has several parameters to determine, comprise ID number, father node, length, unique ID number of every bone definition couples together present node and father node by ID number, the length of every section bone is all inequality, obtains by setting or machine learning.

5, method of following the tracks of according to claim 4 based on the 3 d human motion of many orders camera, it is characterized in that, described every section skeleton is all done rotation and translation around its father node in local coordinate system, wherein: for around the transformation matrix that is rotated of axle that does not overlap with coordinate axis, utilize the rotation of translation and coordinate axis compound and obtain; Rotating the peaceful condition of shifting one's love comprises: the translation skeleton makes it overlap with a coordinate axis that is parallel to this skeleton, for this rotation and translation of object of finishing appointment skeleton is moved back into original position.

6, method of following the tracks of according to claim 1 based on the 3 d human motion of many orders camera, it is characterized in that, described structure human skeleton abstracting method, be meant: judge the effect that human motion is followed the tracks of according to the matching degree of manikin and skeleton observational characteristic, tracking results by preceding frame is also utilized the locus of this frame root node and each section bone is followed the tracks of down frame around the anglec of rotation parameter of its father node human motion, the parameter of first frame is obtained by man-machine interaction or machine learning, front and back two frame parameter variation ranges adopt following method to determine: give every section bone cover a sleeve, the number of the tissue points of calculating in sleeve, as long as it reach maximum and the skeleton that obtains in body surface's point, then stop to calculate, think that this skeleton posture is optimum skeleton posture, has reached the purpose of following the tracks of human motion by the skeleton posture.

7, method of following the tracks of according to claim 1 based on the 3 d human motion of many orders camera, it is characterized in that, the implementation method of described structure Optimum Matching skeleton, be meant: after obtaining the first frame three-dimensional human skeleton, next frame skeleton attitude is adjusted as benchmark with the former frame skeleton, the parameter that changes every frame skeleton posture be every bone around its father node place local coordinate system respectively around X, Y, the Z coordinate axis is done rotational transform, in the middle of the process of this conversion, adopt the probability genetic algorithm to come every bone is carried out iteration around the angle of its father node, up to the human skeleton that obtains human body with interior and can represent human body attitude till, wherein every section bone has individual local optimum, whole skeleton posture reached optimum when all bone all reached local optimum, tracking to this frame finishes, enter the tracking of next frame, to the last till the frame end.

8, according to claim 1 or 6 described methods of following the tracks of based on the 3 d human motion of many orders camera, it is characterized in that, the implementation method of described structure Optimum Matching skeleton, be specially: the definite locus of determining each node of every frame skeleton earlier, the 3 d space coordinate of root node obtains by manual or machine learning, uses the probability genetic algorithm that the anglec of rotation of three coordinate axis of every section bone father node place coordinate system is carried out compound position coding then:

1) determines to carry out the number of the individuality of encoding compound position in conjunction with the limit of people's actual joint motions;

2) parameter that will use every bone is that the anglec of rotation of three coordinate axis of every section bone father node place coordinate system is carried out compound position coding;

3) determine population scale number and iterations, the population scale number is high more, and iterations is many more, and human skeleton posture and original three-dimensional image are just mated more;

4) producing initial population is the set of each root bone around the anglec of rotation of corresponding father node, and it is observed, and utilizes the skeleton observation function to observe the number of the voxel in sleeve;

5) if this quantity increases earlier to tend to be steady again, then change step 6) in the middle of iterative process; Then need to produce new population if increase always, it is rapid to get back to previous step;

6) iteration finishes.

9, method of following the tracks of based on the 3 d human motion of many orders camera according to claim 8 is characterized in that, described skeleton observation function is specific as follows:

$F\left(s\right)=\underset{i=1}{\overset{K}{\mathrm{\&Sigma;}}}\underset{j=1}{\overset{N}{\mathrm{\&Sigma;}}}{x}_j,$ $x_j=\left\{\begin{array}{cc}1& {\mathrm{voxel}}_j\&Element;U\left(s_i\right)\\ 0& {\mathrm{voxel}}_j\&NotElement;U\left(s_i\right)\end{array}\right.$

K represents the bone number that defines, and N represents the quantity of Voxel data mid point,

S=[s ₁, s ₂..., s _k], s represents whole skeleton, s _iExpression bone,

U (s _i) the interior space of expression skeleton sleeve.

Images