CN105787439B

CN105787439B - A Convolutional Neural Network-based Human Joint Localization Method in Depth Image

Info

Publication number: CN105787439B
Application number: CN201610081141.XA
Authority: CN
Inventors: 陈勇杰; 林倞; 王青; 王可泽
Original assignee: Guangzhou New Rhythm Smart Polytron Technologies Inc
Current assignee: Guangzhou New Rhythm Smart Polytron Technologies Inc
Priority date: 2016-02-04
Filing date: 2016-02-04
Publication date: 2019-04-05
Anticipated expiration: 2036-02-04
Also published as: CN105787439A; WO2017133009A1

Abstract

The invention discloses a kind of depth image human synovial localization method based on convolutional neural networks, which is characterized in that including training process and identification process；The step of training process, is as follows: 1) inputting training sample；2) the profound convolutional neural networks of initialization and its parameter, the parameter include the weight and biasing on every layer of side；3) forwards algorithms and backward algorithm are used, the parameter for the convolutional neural networks that training sample learns to construct out is utilized；The step of identification process, is as follows: 4) input test sample；5) position of human joint points therein is returned out to the test sample of input using trained convolutional neural networks.The present invention adds big data using profound convolutional neural networks, is resistant to block, a variety of challenges such as noise, possesses very high accuracy rate；It is calculated simultaneously by parallelization, achievees the effect that human joint points are accurately positioned in real time.

Description

A kind of depth image human synovial localization method based on convolutional neural networks

Technical field

The present invention relates to computer vision, pattern-recognition and field of human-computer interaction, in particular to a kind of to be based on convolutional Neural The depth image human synovial localization method of network.

Background technique

Body pose estimation and motion capture are an important research directions of computer vision field.Its application field packet Include home entertaining, human-computer interaction, action recognition, security system, long-range monitoring, intelligent monitoring, even also patient health nursing Deng.However it is a very challenging job that human posture's estimation is carried out in common RGB image or video.Because It for color, illumination, the factor of natural environment such as blocks and can not accomplish robust, along with human posture too many freedom degree and observation The difference of angle, so that this problem is extremely difficult naturally.

Depth image is a kind of two-dimensional grayscale image, but different from traditional gray level image, each pixel of depth image The gray value reflection of point is the millimeter distance of the corresponding object of point in real space apart from video camera.Compared to traditional Colored two dimensional image, the such environmental effects such as depth image has the characteristics that not to be illuminated by the light, shade, can effectively express true generation The geometry information of object in boundary, therefore have importantly in the research of computer vision and human-computer interaction and application field Position.With popularizing for cheap depth camera, before research and application based on depth image have a vast market and are bright Scape.

Depth image human synovial localization method refers to, in a depth image comprising personage or human body, determines people Body artis position.Here human joint points refer to: the bone of hand, ancon, wrist, shoulder, head, ankle, knee, buttocks et al. Bone joint.It determines that the position of human joint points allows us to parse human skeleton structure, and then simply judges human body Posture, so identify people movement and behavior, this is of great significance for human-computer interaction amusement and computer vision.

The positioning of depth image human synovial is primarily present following difficult point:

1) depth image has the defect that resolution is low, mechanical noise is big.Make the feature of hand-designed to position human body and close Section can not obtain preferable effect.

2) positioning of human synovial because the placement angle of video camera is different, video camera at a distance from personage different, personage from The coverage extent of body is different, and it is extremely difficult to reach accurate robust.

3) there are the constraint relationships between human skeleton joint: when the limb motion of personage, existing between limbs and limbs The constraint relationships such as linkage and braking, and learn and to give expression to this linkage the constraint relationship extremely difficult.

4) positioning in human skeleton joint is difficult to merge with tracking.At present the position of personage and posture positioning both for Individual depth image, reason are the Movement consistency of skeletal joint in the time domain out beyond expression of words.

Above-mentioned difficult point, which to realize, to be carried out the target of human synovial positioning by accurate robust there are also a certain distance, therefore, solution Certainly above-mentioned difficult point is very necessary.

Summary of the invention

The shortcomings that it is a primary object of the present invention to overcome the prior art and deficiency provide a kind of based on convolutional neural networks Depth image human synovial localization method.

In order to reach above-mentioned purpose, the invention adopts the following technical scheme:

Compared with the prior art, the invention has the following advantages and beneficial effects:

1, the present invention, which has relied on, attracts attention at present and has potential technology --- and deep learning constructs profound level Convolutional neural networks, come (these training samples contain multiple angles, the video camera that video camera is put from a large amount of training sample With a variety of coverage extents of a variety of distances on people way and personage itself) in learn effective feature out automatically, without rely on people Hand-designed feature.By learning validity feature out, the artis position of human body is directly returned out.

2, it is consistent to express the movement of skeletal joint in the time domain using Three dimensional convolution layer to convolutional neural networks of the invention Property；It is constrained in top layer using linkage and braking expressed based on the loss function of bone relational tree between skeleton joint etc. Relationship.

Detailed description of the invention

Flow chart Fig. 1 of the invention；

Fig. 2 convolutional neural networks architecture diagram of the present invention；

Fig. 3 human body skeletal joint point schematic diagram of the present invention.

Specific embodiment

Present invention will now be described in further detail with reference to the embodiments and the accompanying drawings, but embodiments of the present invention are unlimited In this.

Embodiment

As shown in Figure 1, the present invention is based on the depth image human synovial localization methods of convolutional neural networks, including trained Journey and identification process；

The step of training process, is as follows:

1) training sample is inputted；

2) the profound convolutional neural networks of initialization and its parameter, the parameter include the weight and biasing on every layer of side；

3) forwards algorithms and backward algorithm are used, the parameter for the convolutional neural networks that training sample learns to construct out is utilized；

The step of identification process, is as follows:

4) input test sample；

5) position of human joint points therein is returned out to the test sample of input using trained convolutional neural networks It sets.

Technical solution of the present invention is further elaborated below with reference to specific technical solution:

1. the framework of convolutional neural networks

The invention proposes a full convolutional neural networks of profound level (as shown in Figure 2), to depth image and depth map As human posture's artis in sequence is estimated.The network is composed in series by 9 convolutional layers, wherein being also interspersed with down-sampled Layer and normalization layer.It will make introductions all round below:

Two-dimensional convolution layer: convolutional layer, which refers to, carries out convolution in two-dimensional space to the image or feature of input, it can be extracted Some important features.Assuming that the width of input picture and height are respectively w and h, the size of three dimensional convolution kernel be w' × h' × M', wherein w', h', m' respectively indicate width, height and port number.A characteristic pattern can be obtained after convolution.Wherein it is located at spy Value at sign figure position (x, y) can be expressed as,

Wherein p_{(x+i)(y+j)(s+k)}Indicate the pixel value of position (x+i, y+j) in (s+k) frame of input, ω_ijkIndicate volume The parameter of product core, b are indicated with biasing relevant to this feature figure.So our available 1 characteristic pattern, each characteristic pattern Size is (w-w'+1, h-h'+1).Since single convolution kernel can only extract a type of feature, we roll up at each layer Lamination introduces multiple convolution kernels and extracts a variety of different features.

Down-sampled layer: down-sampled we are operated using max-pooling.The operation refers to characteristic pattern according to certain strategy (choosing maximum value) carries out down-sampled process.This is a kind of effective procedure being widely used, it can extract holding shape The feature of shape and offset invariance.For one group of characteristic pattern, max-pooling operation is obtained same by down-sampled to them One group of low resolution characteristic pattern of quantity.More, if in a₁×a₂2 × 2 max- is applied on the characteristic pattern of size 2 × 2 not maximum values on overlapping region are extracted in pooling operation, and it is a that we, which will obtain size,₁/2×a₂/ 2 new feature figure.

Correct linear elementary layer (ReLU Nonliearity Layer): the layer is using simple non-linear threshold letter Number, the transformation for only being allowed non-negative signal to pass through input.Assuming that indicating the output of this layer of g, W indicates the weight on this layer of side, a Indicate this layer of input, then we have

G=max (0, W^Ta)

Experiments have shown that in profound convolutional neural networks, the receipts of network when may make trained using the linear elementary layer of correction Hold back speed faster than traditional excitation function.

Full articulamentum: we are added to two layers of full articulamentum in a model, can regard that two-dimensional convolution layer is defeated in front as The perceptron model (hidden layer and logistic regression layer) established on the basis of out.The spy that we will obtain first from M sub-network Sign figure is connected into a long feature vector.The vector indicates the feature being drawn into from range image sequence.It per one-dimensional Element is all connected to all nodes of first full articulamentum (hidden layer), and is further connected to all output units entirely.It is defeated The total 2K of unit out, K indicates the number of bone node here, and the value of output unit is two of bone node on depth image Tie up coordinate position.

Normalization layer: normalization layer refers to concentrating the coordinate manually marked that operation is normalized data.Training one The CNN network of a detection personage, then uses in normalization layer, the target in depth map is cut out to come.It can subtract in this way The interference of few background makes finally to improve the precision of skeleton point detection.

2. the hot map generalization in joint

If given data set is { I_n,L_n, n=1 ..., N, N are the sum of data set sample.Wherein I_nIndicate n-th Image, L_nIndicate the corresponding skeleton point of n-th image, L_n={ l_k, k=1 ..., K, K indicate that shared K is marked Skeleton point, our model setting K is 19, is detailed in Fig. 3.l_k=(x_k,y_k), it is the position of k-th of skeleton point.Assuming that kth The thermal map of a skeleton point is hp_k, l_kIt is mapped in hp_kOn coordinate lh_k=(xh_k,yh_k) it is expressed as follows:

x_k=stride × xh_k+offset (1.1)

y_k=stride × yh_k+offset (1.2)

Wherein, stride expression step-length, offset expression offset, extra setup oneTo determine orange small diamond shape Size.hp_kEach of value indicate the value in I_nIn k-th of skeleton point position probability, value be [0,1].Generate thermal map Algorithm is as follows.

3. the training of model

An i.e. given picture, the network proposed through the invention obtain corresponding K thermal map.It is assumed that this K heat Figure is lined up fixed sequence by human body, can be convenient be compared and learn with true joint thermal map out and really in this way The corresponding prediction thermal map of joint thermal map.In order to normalize the size of input picture, here using the size of first determining thermal map, then calculate The method of the suitable size of input picture out.The size s of thermal map_hp×s_hpDetermined by experience, our model be arranged its for 50 × 50.The then size s of input picture_I×s_IIt is defined as follows:

s_I=(s_hp-1)×stride+offset×2+1 (3.1)

To above formula, since in real joint thermal map training data, the position of human body can be easy to close image border, because This filling for being offset plus size around input picture.The input of our models is image K corresponding with its a true Real joint thermal map exports as the prediction thermal map accordingly to K skeleton point.Doing so, which not only reduces the complexity of model, (keeps away Exempt to train a R-CNN for each thermal map), the weight of each thermal map can also be allowed shared.

Propagated forward:

Each frame image in data set is all propagated into operation along the R-CNN model that we define, first layer is one Layer is normalized, different size of input picture can be normalized to unified size, facilitate processing when subsequent propagation.Then just By full convolutional network as shown in Fig. 2, the size of output is (batch_size, K, S_hp,S_hp), wherein batch_size For the training number of batch training.

Backpropagation:

After the completion of propagated forward, output as described above is obtained.Backpropagation then needs first to find out forward-propagating output Residual error J (ω) between thermal map and real joint thermal map, then acquires it for the gradient of parameter ωAnd using random The algorithm of gradient decline updates ω to minimize residual error, and the loss function J (ω) of residual error is defined as follows.

Wherein | | | |_FFor this black norm of not Luo Beini, Y^predFor the thermal map of prediction, Y^gtFor true joint thermal map

However final effect that only such error back propagation is obtained and bad, reason is that the region of background is remote Much larger than the region of prospect.Therefore we increase a scale factor, i.e., in backpropagation by some proportion by the residual of background Difference is set to 0, and the ratio of foreground and background can be made close.For example, thermal map size is 50 × 50, the small diamond-shaped area of the inside is 5 × 5, then the ratio of foreground and background is then 1:100.The proportional factor r atio that a value is 0.012 is arranged in our model It acts in background, in backpropagation, the ratio regular meeting of foreground and background becomes 1:1.2 from 1:100.

The learning process of model is summarized as algorithm 2:

4. the test of model

A test image is given, is inputted in trained model, can get the thermal map of 19 skeleton points.To each heat Figure, finds out its maximum response, as human body a skeleton point.Finally by formula (1.1) and (1.2), which is become It gains under original image, the coordinate of 19 human body skeleton points can be obtained.Evaluating standard is as follows:

Wherein, pred_coord is the coordinate predicted, and gt_coord is true coordinate, under be designated as the index of skeleton point. What subscript ls (left shoulder) was indicated in denominator is left shoulder, and what rh (right hip) was indicated is right hips, i.e., entirely What denominator indicated is the length of human posture's trunk.This evaluation and test really implies that prediction the distance between coordinate and true coordinate are answered This is less than some proportion of true torso length in image human posture, and it is 20 that r is taken in our model.

The above embodiment is a preferred embodiment of the present invention, but embodiments of the present invention are not by above-described embodiment Limitation, other any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present invention, It should be equivalent substitute mode, be included within the scope of the present invention.

Claims

1. a depth image human body joint positioning method based on convolutional neural network, is characterized in that, comprises training process and identification process;

The steps of the training process are as follows:

1) Input training samples;

2) Initialize a deep convolutional neural network and its parameters, including the weights and biases of each layer edge;

3) Using the forward algorithm and the backward algorithm, and using the training samples to learn the parameters of the constructed convolutional neural network;

The steps of the identification process are as follows:

4) Input test samples;

5) Use the trained convolutional neural network to return the position of the human body joint points to the input test sample;

In step 3), the generation step of the joint heat map is also included, specifically:

Let the given dataset be {I _n ,L _n },n=1,...,N, where N is the total number of samples in the dataset, where I _n represents the nth image, and L _n represents the nth image corresponding to , L _n ={l _k },k=1,...,K, K represents a total of K marked human skeleton points; l _k =(x _k ,y _k ), which is the kth The position of the skeleton point, assuming that the heat map of the kth skeleton point is hp _k , the coordinates lh _k =(xh _k ,yh _k ) of l _k mapped on hp _k are expressed as follows:

x _k = stride×xh _k +offset (1.1)

y _k = stride×yh _k +offset (1.2)

Among them, stride represents the step size, offset represents the offset, and an additional l is set to determine the size of the small orange diamond, and each value in hp _k represents the probability of the value at the position of the _kth bone point in In, take The value is [0,1].

2. the depth image human body joint localization method based on convolutional neural network according to claim 1, is characterized in that, the training sample in described step 1) is the original depth image that contains character captured by the depth camera of free angle and a collection of annotations.

3. the depth image human body joint positioning method based on convolutional neural network according to claim 1, is characterized in that, the convolutional neural network in described step 2) has deep structure, and is composed of convolutional layer, descending The sampling layer, the rectified linear unit layer, and the fully connected layer are stacked and normalized through the normalization layer. The top layer of the convolutional neural network directly outputs the positions of the human joint points.

4. The depth image human body joint positioning method based on convolutional neural network according to claim 3, is characterized in that, described convolutional layer refers to that the input image or feature is convolved in two-dimensional space, and extracts important parameters. feature;

The down-sampling layer uses a max-pooling operation, which refers to the process of down-sampling the feature map according to a set strategy, and is used to extract features that maintain shape and offset invariance;

The rectifying linear unit layer adopts a simple nonlinear threshold function, and performs a transformation on the input that only allows non-negative signals to pass through;

In the fully connected layer, the feature maps obtained from the M sub-networks are first concatenated into a long feature vector, which represents the features extracted from the depth image sequence, and each dimension element of it is connected to the first All nodes of a fully connected layer are further fully connected to all output units. There are 2K output units in total, where K represents the number of skeleton nodes, and the value of the output unit is the two-dimensional coordinate position of the skeleton node on the depth image. ;

The normalization layer is to perform a normalization operation on the manually marked coordinates in the dataset.

5. the depth image human body joint positioning method based on convolutional neural network according to claim 1, is characterized in that, in step 3), when training, suppose that K heat maps are arranged in a fixed order by human body parts, for Compare the joint heatmap and learn the predicted heatmap corresponding to the real joint heatmap. In order to normalize the size of the input image, first determine the size of the heatmap, and then calculate the appropriate size of the input image; the size of the heatmap s _hp × s _hp is determined by experience, then the size of the input image s _I ×s _I is defined as follows:

s _I = (s _hp -1)×stride+offset×2+1 (3.1)

For the above formula, since in the real joint heat map training data, the human body will easily be close to the edge of the image, so add a padding of size offset around the input image, and the input of the model is the image and its corresponding K real Joint heatmap, the output is the corresponding predicted heatmap for K bone points.

6. the depth image human body joint positioning method based on convolutional neural network according to claim 5, is characterized in that, forward algorithm is specifically:

Each frame of image in the data set is propagated along the defined R-CNN model. The first layer is a normalization layer, which can normalize input images of different sizes to a uniform size, which is convenient for subsequent propagation. processing, and then through the fully convolutional network, the output size is (batch_size, K, S _hp , S _hp ), where batch_size is the number of training batches;

The backward algorithm is specifically:

Backpropagation needs to first obtain the residual J(ω) between the heat map output by the forward propagation and the real joint heat map, and then obtain its gradient to the parameter ω. And the stochastic gradient descent algorithm is used to update ω to minimize the residual, and the loss function J(ω) of the residual is defined as follows;

where ||·|| _F is the Frobenius norm, Y ^pred is the predicted heatmap, and ^Ygt is the real joint heatmap.

7. The convolutional neural network-based depth image human body joint positioning method according to claim 1, wherein in the step 4), the test sample is the original depth image captured by a free-angle depth camera.

8. the depth image human body joint positioning method based on convolutional neural network according to claim 1, is characterized in that, in step 5), the concrete method of returning the position of human body joint point wherein is:

Given a test image and input it into the trained model, the heat map of the skeleton points can be obtained. For each heat map, find the maximum response value, which is a skeleton point of the human body, and finally pass the formula (1.1) With (1.2), the coordinates are transformed back to the original image, and the coordinates of the body skeleton points can be obtained. The evaluation criteria are as follows:

Among them, pred_coord is the predicted coordinate, gt_coord is the real coordinate, the subscript is the index of the skeleton point, the subscript ls in the denominator represents the left shoulder, and rh represents the right hip, that is, the whole denominator represents the body posture of the torso. length.