CN115273143A - A head pose estimation method and system - Google Patents

Abstract

The invention provides a head posture estimation method and a system, comprising the following steps: determining an image containing a human face; inputting the image into a pre-trained hierarchical prediction network, and predicting to obtain a pitch angle, a yaw angle and a roll angle of the face posture orientation; the method comprises the following steps: the system comprises a backbone network, a characteristic pyramid network, a dimension reduction module and a hierarchical prediction module; the system comprises a backbone network, a characteristic pyramid network, a dimension reduction module and a dimension extraction module, wherein the backbone network is used for extracting image space features with different sizes, the characteristic pyramid network is used for fusing the image space features with different sizes to obtain fused features, and the dimension reduction module is used for performing three-dimension reduction on the fused features to obtain three-dimension space features of images; the hierarchical prediction module comprises: three full-link layers; the three full-connection layers respectively predict the spatial features of three dimensions, and each full-connection layer predicts an angle of the face pose orientation, so that the image regions focused by the three angles of the face pose orientation predicted by the hierarchical prediction network are different, and the mutual interference among the three angle predictions is reduced.

Description

A head pose estimation method and system

technical field

The present invention belongs to the field of head posture detection, and more specifically, relates to a head posture estimation method and system.

Background technique

Head posture detection technology has a wide range of applications, such as: fatigue detection, automatic driving, etc. On the one hand, although the use of depth images can achieve very good results in head pose estimation, for RGB images, the head pose estimation method still has the problem of discontinuous angle prediction, which limits the application of head pose estimation. .

In a real environment, characters often have large-scale facial occlusion and large-angle head deflection; the background environment they are in, and the brightness and darkness of the light are also very different. The method based on traditional machine learning is difficult to detect the head of the person in this situation, and it is not robust to people with different identities, and cannot complete the normal head pose estimation task.

Compared with general machine learning methods, deep learning methods have better performance in the image field, and are more suitable for head pose estimation in real scenes than machine learning methods, specifically in the following three points: (1) Different identities People exhibit good robustness; (2) are not sensitive to background changes of people; (3) head pose estimation tasks can be performed from a single image, making real-time detection possible.

At the same time, the deep learning method also has certain defects in the task of head pose estimation: (1) In the existing methods of head pose estimation, the parameter adjustments of the three angles interfere with each other, and the prediction effect of the model is difficult to balance; (2) ) using a simple addition of the cross-entropy loss function and the mean square error loss function, resulting in discontinuities in the angle predictions.

Contents of the invention

Aiming at the defects of the prior art, the purpose of the present invention is to provide a method and system for estimating head pose, aiming to solve the problem that the parameter adjustments of the three angles of the existing head pose estimation interfere with each other and the angle prediction is discontinuous.

In order to achieve the above object, in a first aspect, the present invention provides a head pose estimation method, comprising the following steps:

Identify images that contain human faces;

The image is input to the pre-trained layered prediction network, and the pitch angle, yaw angle and roll angle of the attitude of the face are predicted to estimate the head pose of the face; the layered prediction network includes: a backbone network , a feature pyramid network, a dimensionality reduction module, and a hierarchical prediction module; the backbone network is used to extract image space features of different sizes, and the feature pyramid network is used to fuse image space features of different sizes to obtain fusion features, and the reduction The dimension module is used to perform dimensionality reduction in three different dimensions on the fusion feature to obtain spatial features of the image in three dimensions, and different dimensions correspond to different image channel numbers; the layered prediction module includes: three fully connected layers; The three fully connected layers predict the spatial features of the three dimensions respectively, and each fully connected layer predicts an angle of the orientation of the face pose, so that the layered prediction network predicts the orientation of the face pose toward the three dimensions. Each angle focuses on different image areas, reducing the mutual interference between the three angle predictions; the size is in pixels.

In an optional example, the loss function in the training process of the hierarchical prediction network adopts a self-adjusting loss limit coefficient, so that when the average absolute error of the prediction angle of the three fully connected layers is less than a threshold, the correction is made by the prediction angle The cross-entropy loss item is greater than the mean square error loss item of the prediction angle, which brings about the loss size reversal problem, and when the average absolute error of the prediction angle of the three fully connected layers is not less than the threshold value, the increase caused by the above two loss items The error penalty makes the hierarchical prediction network converge at a faster speed during the training process; when the average absolute error of the prediction angles of the three fully connected layers is less than the threshold, the error penalty of the hierarchical prediction network is the first penalty, and three When the average absolute error of the prediction angle of the fully connected layer is not less than the threshold, the error penalty of the layered prediction network is the second penalty, and the correction loss size reverse problem means that the first penalty is controlled to be less than the second penalty, so that the layered prediction network can be trained and learned normally .

In an optional example, the backbone network includes four residual blocks; the face image is sequentially processed by the four residual blocks, and four image space features with decreasing sizes are sequentially obtained;

The above four size-decreasing image space features are fused by the feature pyramid network. The fusion strategy is to first fuse the first size space feature with the second size space feature, and combine the first size space feature with the third size space feature. The spatial features are fused to obtain the new second-dimensional spatial features and the new third-dimensional spatial features, and then the new second-dimensional spatial features are fused with the new third-dimensional spatial features, and the new third-dimensional spatial features are fused. The second dimension space feature is fused with the fourth dimension space feature to obtain the updated third dimension space feature and the new fourth dimension space feature, and finally the updated third dimension space feature and the new dimension space feature The fourth size space feature is fused to obtain the fusion feature of the fourth size; wherein, the size from the first size to the fourth size decreases step by step;

The dimensionality reduction module includes three convolution kernels; the fusion features of the fourth size are sequentially processed by the three convolution kernels, and each convolution kernel performs a dimensionality reduction on the input image features, and successively obtains different sizes. The spatial characteristics of the three dimensions change and the number of channels decreases step by step.

In an optional example, the adjustment formulas of the hierarchical prediction network for the three prediction angles are:

以及

分别表示俯仰角、偏航角以及翻滚角的预测值；K₁、K₂以及K₃分别为三个卷积核的权重因子；Γ₁、Γ₂以及Γ₃为所述融合特征经过降维模块三个卷积核得到的三种维度的空间特征；in,

as well as

respectively represent the predicted values of pitch angle, yaw angle and roll angle; K ₁ , K ₂ and K ₃ are the weight factors of the three convolution kernels respectively; Γ ₁ , Γ ₂ and Γ ₃ are the dimensionality reduction of the fusion features The three-dimensional spatial features obtained by the three convolution kernels of the module;

The relationship between Γ ₁ , Γ ₂ , and Γ ₃ satisfies the following formula:

Among them, W ₁ is the feedback parameter from the first convolution kernel to the second convolution kernel of the dimensionality reduction module, W ₂ is the feedback parameter from the second convolution kernel to the third convolution kernel of the dimensionality reduction module, b ₄ is the new bias term brought by the dimensionality reduction from the first convolution kernel to the second convolution kernel, and b ₅ is the new bias term brought by the dimensionality reduction from the second convolution kernel to the third convolution kernel.

为：In an optional example, the loss function of the hierarchical prediction network

for:

是分层预测网络对头部姿态预测得到的值，y是图像中人脸头部姿态的真实值，β为损失限制系数，是由均方差损失和交叉熵损失的同大同小关系构建得出，k为角度类别数，σ是sigmod函数，L_mse代表均方差损失，Y_ic代表根据角度类别所形成的one-hot编码，

是预测角度所属的类别。in,

is the value predicted by the hierarchical prediction network for the head pose, y is the real value of the head pose of the face in the image, and β is the loss restriction coefficient, which is constructed by the relationship between the mean square error loss and the cross entropy loss , k is the number of angle categories, σ is the sigmod function, L _mse represents the mean square error loss, Y _ic represents the one-hot code formed according to the angle category,

is the category to which the predicted angle belongs.

In a second aspect, the present invention provides a head pose estimation system, comprising:

A human face image determining unit, configured to determine an image containing a human face;

The head posture estimation unit is used to input the image into the pre-trained layered prediction network, and predicts the pitch angle, yaw angle and roll angle towards the face posture to estimate the head posture of the human face; The layered prediction network includes: a backbone network, a feature pyramid network, a dimensionality reduction module, and a layered prediction module; the backbone network is used to extract image space features of different sizes, and the feature pyramid network is used to fuse image space features of different sizes , to obtain the fusion feature, the dimensionality reduction module is used to perform dimensionality reduction in three different dimensions on the fusion feature, to obtain the spatial features of the image in three dimensions, and different dimensions correspond to different image channel numbers; the layered prediction module Including: three fully connected layers; the three fully connected layers respectively predict the spatial features of the three dimensions, and each fully connected layer predicts an angle towards the orientation of the face posture, so that the layered prediction The network predicts the face poses of the three angles to focus on different image areas, reducing the mutual interference between the three angle predictions; the size is in pixels.

In an optional example, the loss function used in the layered prediction network training process used by the head pose estimation unit adopts a self-adjusting loss limit coefficient, so that the average absolute error of the three fully connected layer prediction angles is less than a threshold When the cross entropy loss item of the prediction angle is greater than the mean square error loss item of the prediction angle, the loss size reversal problem is corrected, and when the average absolute error of the prediction angle of the three fully connected layers is not less than the threshold value, the above two The error penalty brought by the loss item makes the hierarchical prediction network converge at a faster speed during the training process; when the average absolute error of the three fully connected layer prediction angles is less than the threshold, the error penalty of the hierarchical prediction network is The first penalty, when the average absolute error of the prediction angles of the three fully connected layers is not less than the threshold, the error penalty of the layered prediction network is the second penalty, and the correction loss size reverse problem means that the first penalty is controlled to be less than the second penalty, and the score is guaranteed The layer prediction network can train and learn normally.

In an optional example, the backbone network used by the head pose estimation unit includes four residual blocks; the face image is sequentially processed by the four residual blocks, and four kinds of image space features with decreasing sizes are sequentially obtained. ; The image space features of the above four kinds of decreasing sizes are fused by the feature pyramid network, and the fusion strategy is to first fuse the first size space feature with the second size space feature, and combine the first size space feature with the third The dimension space features are fused to obtain the new second dimension space feature and the new third dimension space feature, and then the new second dimension space feature is fused with the new third dimension space feature, and the new The second dimension space feature is fused with the fourth dimension space feature to obtain the updated third dimension space feature and the new fourth dimension space feature, and finally the updated third dimension space feature and the new dimension space feature The fourth size space feature is fused to obtain the fusion feature of the fourth size; wherein, the size of the first size to the fourth size decreases step by step; the dimensionality reduction module includes three convolution kernels; The fusion features of the fourth size are sequentially processed by the three convolution kernels, and each convolution kernel performs a dimensionality reduction on the input image features, and sequentially obtains a three-dimensional space with a constant size and a gradually decreasing number of channels. feature.

其中，

以及

分别表示俯仰角、偏航角以及翻滚角的预测值；K₁、K₂以及K₃分别为三个卷积核的权重因子；Γ₁、Γ₂以及Γ₃为所述融合特征经过降维模块三个卷积核得到的三种维度的空间特征；Γ₁，Γ₂，Γ₃之间的关系满足如下公式：

其中，W₁是降维模块第一个卷积核向第二个卷积核的反馈参数，W₂是降维模块第二个卷积核向第三个卷积核的反馈参数，b₄是经第一个卷积核向第二个卷积核降维带来的新偏差项，b₅是经第二个卷积核向第三个卷积核降维带来的新偏差项。In an optional example, the adjustment formula of the hierarchical prediction network used by the head pose estimation unit for the three predicted angles is:

in,

as well as

respectively represent the predicted values of pitch angle, yaw angle and roll angle; K ₁ , K ₂ and K ₃ are the weight factors of the three convolution kernels respectively; Γ ₁ , Γ ₂ and Γ ₃ are the dimensionality reduction of the fusion features The three-dimensional spatial features obtained by the three convolution kernels of the module; the relationship between Γ ₁ , Γ ₂ , and Γ ₃ satisfies the following formula:

Among them, W ₁ is the feedback parameter from the first convolution kernel to the second convolution kernel of the dimensionality reduction module, W ₂ is the feedback parameter from the second convolution kernel to the third convolution kernel of the dimensionality reduction module, b ₄ is the new bias term brought by the dimensionality reduction from the first convolution kernel to the second convolution kernel, and b ₅ is the new bias term brought by the dimensionality reduction from the second convolution kernel to the third convolution kernel.

为：In an optional example, the loss function of the hierarchical prediction network used by the head pose estimation unit

for:

是分层预测网络对头部姿态预测得到的值，y是图像中人脸头部姿态的真实值，β为损失限制系数，是由均方差损失和交叉熵损失的同大同小关系构建得出，k为角度类别数，σ是sigmod函数，L_mse代表均方差损失，Y_ic代表根据角度类别所形成的one-hot编码，

是预测角度所属的类别。in,

is the value predicted by the hierarchical prediction network for the head pose, y is the real value of the head pose of the face in the image, and β is the loss restriction coefficient, which is constructed by the relationship between the mean square error loss and the cross entropy loss , k is the number of angle categories, σ is the sigmod function, L _mse represents the mean square error loss, Y _ic represents the one-hot code formed according to the angle category,

is the category to which the predicted angle belongs.

Wherein, k is the number of angle categories. Specifically, in the embodiment of the present invention, -99° to 99° is divided into 66 angle intervals according to 3°, and the corresponding number of angle categories is 66. In addition, those skilled in the art may divide the angles into different categories according to actual needs, which is not further limited in the present invention.

Generally speaking, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:

The present invention provides a method and system for estimating head pose, which regards head pose estimation as three branches of the same task, integrates feature pyramid and multi-task convolution ideas, and reduces The mutual interference between the three angle adjustments is eliminated, so that the results of head pose estimation have smaller deviations. The traditional method adopts the simple addition of cross entropy and mean square error loss to carry out the training of head pose estimation, but the present invention optimizes the loss on the basis of analyzing the disadvantages of the traditional loss function, and solves the problem caused by the discontinuity of the loss function itself. The resulting intermittent problem of angle estimation loss further improves the results of head pose estimation. This method is compatible with the latest head pose estimation method based on rotation matrix, which provides the possibility to further improve the accuracy of head pose estimation in the future.

Description of drawings

FIG. 1 is a flowchart of a head pose estimation method provided by an embodiment of the present invention;

FIG. 2 is a block diagram of the implementation of the head pose estimation method provided by the embodiment of the present invention;

3 is a frame diagram of a head pose estimation model proposed by an embodiment of the present invention;

Fig. 4 is an explanatory diagram of the angle prediction discontinuity problem provided by the embodiment of the present invention;

Fig. 5 is a heat map of attention differences from different angles proposed by the embodiment of the present invention;

FIG. 6 is a schematic diagram of the additional attention provided by the embodiment of the present invention;

FIG. 7 is an effect diagram of head pose estimation provided by an embodiment of the present invention;

FIG. 8 is an architecture diagram of a head pose estimation system provided by an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Fig. 1 is a flow chart of the head pose estimation method provided by the embodiment of the present invention, as shown in Fig. 1, comprising the following steps:

S101, determine an image containing a human face;

S102, inputting the image into a pre-trained layered prediction network, predicting the pitch angle, yaw angle and roll angle of the attitude of the face to estimate the head pose of the face; the layered prediction network includes: A backbone network, a feature pyramid network, a dimensionality reduction module, and a hierarchical prediction module; the backbone network is used to extract image space features of different sizes, and the feature pyramid network is used to fuse image space features of different sizes to obtain fusion features, so The dimensionality reduction module is used to perform dimensionality reduction in three different dimensions on the fusion feature to obtain spatial features in three dimensions of the image, and different dimensions correspond to different numbers of image channels; the layered prediction module includes: three full connections layer; the three fully connected layers predict the spatial features of the three dimensions respectively, and each fully connected layer predicts an angle towards the attitude of the human face, so that the layered prediction network predicts the orientation of the human face attitude The image areas concerned by the three angles are different, so as to reduce the mutual interference between predictions from the three angles; the size is in pixels.

Specifically, the present invention adopts the following technical solutions: provide a head pose estimation method for loss adaptive adjustment under hierarchical prediction, wherein, hierarchical prediction refers to the use of different network layers to separate the prediction tasks of three angles; loss Adaptive adjustment is to add restrictions according to the loss function itself. From the perspective of the loss function, the problem of discontinuity in angle prediction is solved. The specific steps are as follows:

(1) Cut a single RGB image of a person, or an image in a standard data set, for face cropping. The size of the cropped image is 224×224. Since the data preprocessing method is relatively common and the principle is relatively simple, here not to repeat;

(2) In the model, feature pyramids are used to fuse image features of different scales extracted from different network layers, and a dynamic adaptive spatial feature fusion strategy is adopted to automatically assign the weight of the fused features. Using ResNet 50 as the backbone network, combined with the feature pyramid strategy, only the features spanning two levels of scales are fused during the fusion process, and the fusion processing is not performed if the ratio exceeds this ratio. After merging the extracted image features of different sizes, a fused summary feature is obtained, and the feature is reduced three times to form three features with the same spatial resolution but different numbers of channels, which are used for the heads of the three angles respectively. pose estimation task.

(3) Connect the three features obtained in step (2) with the fully connected layer respectively, accumulating three fully connected layers, and form a one-to-one connection relationship between the feature and the fully connected layer. At this point, the model has space for separate adjustments for the three angles of yaw, pitch, and roll, and the interaction between the adjustments of the three angles is reduced. yaw represents the angle of rotation around the y-axis, known as the yaw angle; pitch represents the angle of rotation around the x-axis, known as the pitch angle; roll represents the angle of rotation around the z-axis, known as the roll angle.

(4) The three branches formed in step (3) are equipped with additional attention mechanisms, so that the features that the model pays attention to in parameter mediation are more concentrated and have identity robustness. External attention makes the features of the model focus on the same features of people with different identities, which makes the predictions from the three perspectives form their own unique feature focus.

(5) The three features extracted by external attention in step (4) are trained using cross-entropy loss and mean square error loss. In particular, the present invention proposes a dynamic self-adjusting loss constraint item, which solves the discontinuous problem of angle prediction existing in the traditional training process. , the mean square error is used to constrain the cross-entropy loss, so that the prediction loss of the model during training and the generated real angle loss maintain the same increase and decrease trend.

The ideas involved in each step of the present invention are generally introduced as follows: firstly, the face image is obtained by cropping, the interference of irrelevant background factors is removed, and the reduction of the image size also reduces the calculation burden of the model. Secondly, through the feature pyramid fusion strategy, the features of different scales can play a certain role. The fused features have both details and overall aspects, eliminating the disadvantages of high-level features that tend to focus on the whole due to too deep convolutional layers. Furthermore, the stepped head pose prediction formed by feature dimensionality reduction takes into account the advantages of traditional feature fusion and multi-task prediction, so that the three angles can achieve good coordination on one model. At the same time, the attention mechanism makes the features extracted by the model universal. Finally, the optimized loss function is used to make the overall head pose estimation present a continuity, and the accuracy of the head pose estimation is greatly improved.

After the above five steps, the head pose estimation method proposed by the present invention solves the problem of angle interaction and discontinuous angle prediction in the traditional head pose prediction process, and the additional computing power brought by the feature pyramid Consumption is also mitigated by dimensionality reduction operations. After verification on the standard data set, the head pose estimation method proposed by the present invention is robust to people with different identities or different poses of the same person.

The present invention provides a head pose estimation method for loss adaptive adjustment under hierarchical prediction, and its specific implementation steps are as follows:

Fig. 2 is a block diagram of the implementation of the head pose estimation method for loss adaptive adjustment under hierarchical prediction according to the present invention. As shown in Fig. 2, the head pose estimation method proposed by the present invention as a whole includes the following modules: (1) Image input module; (2) Feature fusion module; (3) Hierarchical prediction module; (4) Loss limitation module. The specific operation steps involved in each module are as follows:

1. Image input module: First, obtain the image or video of the person whose head posture is to be detected, and perform preprocessing operations. The specific operation process includes but is not limited to: performing face cropping to obtain the head image of the person with the background removed. Resize the picture, and adjust the size of the picture to 224×224 pixels; arrange the pictures in the video according to the frame sequence (fps=60), and arrange the pictures in the picture set in order; it should be noted that the size of the picture is adjusted There is no specific sequence requirement for the two steps of sorting pictures. After the pre-processing is completed, all pictures need to be randomly occluded. The purpose of this processing is to prevent the model from paying too much attention to local features during the training process, resulting in a decrease in the versatility of the model. The output of the image input module is a batch of processed normalized images of human heads.

2. Feature fusion module: As shown in Figure 2, the feature fusion module is located below the image input module, and is used to receive the standardized image output by the image input module and input the fused features to the hierarchical prediction module. The detailed schematic diagram of the feature fusion module can be seen in Figure 3. The operation steps included are: extracting features from the standard image through the backbone network Resnet-50, and only adopting the downsampling strategy for the features extracted by different blocks. In terms of feature fusion, for the case where the spatial scale ratio is 2:1, the present invention uses a convolutional layer with a step size of 2 and a convolution kernel size of 3×3 to ensure that the spatial scale is consistent; for a spatial scale ratio of 4:1 In this case, the present invention first uses maxpooling with a step size of 2 for maximum pooling, and then uses a convolutional layer with a step size of 2 and a convolution kernel size of 3×3 to ensure consistent spatial scales. However, for the spatial scale ratio of 8:1, the present invention does not adopt a spatial fusion strategy for it because the features differ too much. Using S to represent each stage, the above process of feature fusion can be described as:

Among them, Sj|j=3, 4 means the last two block stages, →j means fusion based on the feature space scale of the current block, and γ is the fusion weight. When j=1 or 2, the value corresponding to γ ₂ or γ ₃ is 0, that is, only two stages of feature fusion are performed at this time. Meanwhile, the present invention forces γ ₁ +γ ₂ +γ ₃ =1|γ ₁ ,γ ₂ ,γ ₃ ∈[0,1]. In order to achieve this goal, the present invention uses three 1×1 convolutional layers to calculate weights, the formula is as follows:

为第一尺度特征对应的权值，

为第二尺度特征对应的权值，

为第三尺度特征对应的权值，γ₁为经过类似sigmod加权计算后得到的占比，最终使得γ₁+γ₂+γ₃＝1。

is the weight corresponding to the first scale feature,

is the weight corresponding to the second scale feature,

is the weight corresponding to the third-scale feature, and γ ₁ is the proportion obtained after a similar sigmod weighted calculation, which finally makes γ ₁ +γ ₂ +γ ₃ =1.

After feature fusion, the model not only retains small features, but also pays attention to the overall part of the image. In the next stage, this fused feature will be further dimensionally reduced to form a ladder-like hierarchical prediction.

In a specific embodiment, an image with a size of 224×224 is input into the convolutional neural network, and after a convolution operation with a convolution kernel size of 3×3, the spatial size of the image becomes 112×112. Afterwards, when passing through the four blocks in the backbone network ResNet-50, image space features with sizes of 56×56, 28×28, 14×14 and 7×7 are formed. The four-dimensional spatial features are fused using the feature pyramid, and the spatial size needs to be kept consistent during fusion. The fusion strategy is: for a 56×56-sized spatial feature, a layer of maximum pooling is used to make the spatial feature a 28×28 size , and then undergo a 3×3 convolution to make the size into a 14×14 size, and then merge with these two dimensions respectively, and the rest of the fusion operations can be deduced by analogy. For features spanning two scales, such as 112×112 and 7×7 features, fusion is not performed. After the feature pyramid, the fused image feature size is 7×7×2048. At this time, the number of channels is reduced by using the 1×1 convolution kernel, that is, the number of channels is reduced to 7×7×1024, which is called dimensionality reduction. Layer 1 (dw1), and then perform dimensionality reduction processing on dw1 to obtain dimensionality reduction layer 2 (dw2), and then dimensionality reduction to obtain dimensionality reduction layer 3 (dw3). At this time, the features of the three dimensions have the same spatial resolution ratio and are all 7×7 in size, but the number of channels is different.

3. Hierarchical prediction module: As shown in Figure 2, the main function of the hierarchical prediction module is to form three branches of predictions from three perspectives through dimensionality reduction, and the predictions of each branch do not interfere with each other. Before elaborating the hierarchical prediction function proposed by the present invention, it is necessary to explain the methods commonly used at present: the traditional method regards the prediction of the three angles of the head posture as three branches of the same task, and they completely share the same The network layer, which increases the burden of the model, as shown in Figure 4, AP is Avg pooling, when the model adjusts the model parameters according to the loss feedback of other angles, the prediction result of an angle with a smaller prediction loss may be worse because the model must balance between these three angles of adjustment. MAE in Fig. 4 means mean absolute error.

4. In short, the simultaneous estimation of three angles limits the performance of the model relative to head pose estimation from a single angle. In the traditional head pose estimation task, the prediction of the three angles can be described as the following formula:

θ和ψ分别表示对yaw，pitch，roll的预测值。假设一个图像的预测损失是

由于网络层是共享梯度反向传播，调整后的预测损失变为

虽然总的预测损失降低了，但它并不是偏航的最佳模型。经过本发明提出的分层预测结构之后，三种角度的调节公式更改为：Among them, K represents different weights, Γ is the feature extracted by the convolutional layer, b is the bias factor, using

θ and ψ represent the predicted values of yaw, pitch and roll, respectively. Suppose the prediction loss for an image is

Since the network layers are shared gradient backpropagation, the adjusted prediction loss becomes

Although the overall prediction loss is reduced, it is not the best model for yaw. After the hierarchical prediction structure proposed by the present invention, the adjustment formulas of the three angles are changed to:

Among them, Γ ₁ , Γ ₂ , and Γ ₃ are the features obtained after dimensionality reduction of the three dimensionality reduction layers dw1, dw2, and dw3 obtained through dimensionality reduction of the above fusion features. The correlation between Γ ₁ , Γ ₂ , and Γ ₃ is as follows, where W ₁ and W ₂ are new convolution parameters, and b ₄ and b ₅ are new bias items brought about by dimensionality reduction:

The present invention regards head pose estimation as three tasks, provides additional adjustment space for model parameter adjustment, and the prediction order of angles is determined by the distribution of the number of samples in the data set. As shown in FIG. 5 , (a) in the figure represents the attention feature parts of various angles obtained by the traditional method, and (b) represents the attention feature parts of three angles obtained by the hierarchical prediction method proposed by the present invention. It can be seen that after stratification, the areas of concern of the three perspectives are no longer the same, which means that the hierarchical prediction strategy has played a role.

Afterwards, the present invention adds a layer of external attention to the prediction of each angle. The working principle of the external attention mechanism is shown in Figure 6. By continuously extracting the common features between the pictures participating in the training, these common features correspond to The weights are constantly increasing, while the weights of other parts are relatively weakened. The hierarchical prediction module takes a batch of features obtained from the backbone network as input, and then uses a 1×1 convolutional layer to scale the channel to reduce the computational burden. After a layer of external attention mechanism, the 1×1 convolution The layer restores the number of channels, and finally outputs to the loss limit module.

4. Loss limitation module: Before explaining the problem solved by the method of the present invention, it is necessary to explain the loss function prediction discontinuity problem caused by the traditional method, as shown in Figure 4, when the real head posture is [6.1°, When -3.2°, -15°] and prediction angles are [5.9°, -1.9°, -9.9°], because the classification loss is greater than the regression loss, the traditional loss function incorrectly reverses the relationship between the actual loss of yaw and pitch. In addition, the traditional loss function also leads to the problem of unbalanced loss on both sides of the angle classification line, intermittent loss and wrongly inverted loss function make the model difficult to learn. Another simple example to illustrate the problem of imbalanced loss at both ends of the classification. Set the real angle to [0°, 3°, 5°] and the predicted angle to [1°, 3.5°, 7°], and set the head angles between (-99°, 99°) at intervals of 3° , divided into 66 categories. When the prediction loss is within 1°, there are two cases of angle prediction: inter-class loss and intra-class loss. When the prediction loss is an intra-class loss, the cross-entropy loss is smaller, and the total loss is consistent with the true loss. But when the prediction loss is between-class loss, since the exponential term of the mean square error is 2, the cross entropy loss will be greater than the mean square loss at this time, which will cause the total loss to be opposite to the true loss trend, making the model difficult to learn. The traditional loss function approach can be described as:

是预测角度所属的类别，σ表示softmax，L_ce表示交叉熵损失，L_mse代表均方差损失。Among them, k is the number of categories, and Y _ic represents the one-hot code formed according to the angle category, which is 0 or 1, indicating whether the classification is correct,

is the category to which the prediction angle belongs, σ represents softmax, L _ce represents cross entropy loss, and L _mse represents mean square error loss.

经更新后的头部姿态估计损失函数如下：Considering the synergy between the two losses, the present invention sets additional constraints on the classification loss:

The updated head pose estimation loss function is as follows:

After the loss limit, the loss item β∈[0,1] is also added to the backpropagation gradient. When the real loss is small, the penalty obtained is small. In the above example, the cross-entropy loss of the pitch angle is reduced to 1/5 by β, which resets the total loss of the model to the same trend as the real loss, thus solving the angle prediction brought by the traditional loss function Inconsistencies. The present invention increases the error penalty when the angle loss is above 1°, so as to speed up the convergence speed of the model. The input image goes through four prediction modules to complete a round of training process. The model adjusts the parameters through the back propagation mechanism, so that the angle prediction is continuously perfected.

In order to better explain the positioning and detection system combining position information and head posture provided by the present invention, the following specific description will be given in conjunction with embodiments.

Fig. 7 is a schematic diagram of head pose estimation in a complex environment provided by an embodiment of the present invention. Fig. 7 includes not only the case of large-angle deflection, but also the case of head being blocked by an obstacle. At the same time, use the method of the present invention and the traditional method to predict, and compare the obtained results. As shown in Figure 7, when the head posture is deflected at a large angle or there is an occluder in the head posture, the traditional head posture estimation method Compared with the method proposed in the present invention, the average loss of angle prediction is reduced by more than 10°, and the prediction for each angle is close to the real value. This shows that, compared with traditional methods, the head pose estimation method proposed in the present invention has robustness in complex scenes, which proves that the head pose angle hierarchical prediction strategy and loss function self-adjustment strategy involved in the present invention are Effective.

In Fig. 7, Ground truth is the real data of the head pose angle, HopeNet is an existing method, and its full name is: fine-grained head pose estimation without key points; TPL-net is the head pose estimation method of the present invention, and this The full name of the layered prediction network used in the invention is: layered prediction and loss limit network, Tiered prediction with loss limit network.

Fig. 8 is an architecture diagram of the head pose estimation system provided by the embodiment of the present invention, as shown in Fig. 8, including:

A human face image determination unit 810, configured to determine an image comprising a human face;

The head posture estimation unit 820 is used to input the image into the pre-trained layered prediction network to predict the pitch angle, yaw angle and roll angle of the face posture orientation, so as to estimate the head posture of the human face; The layered prediction network includes: a backbone network, a feature pyramid network, a dimensionality reduction module, and a layered prediction module; Fusion to obtain fusion features, the dimensionality reduction module is used to perform dimensionality reduction in three different dimensions on the fusion features to obtain spatial features in three dimensions of the image, and different dimensions correspond to different numbers of image channels; the layered prediction The module includes: three fully connected layers; the three fully connected layers predict the spatial features of the three dimensions respectively, and each fully connected layer predicts an angle towards the orientation of the face, so that the layered The prediction network predicts that the three angles of the face pose are different in the image area of interest to reduce the mutual interference between the three angle predictions; the size is in pixels.

It can be understood that, for the detailed function implementation of each unit in FIG. 8 , reference may be made to the introduction in the foregoing method embodiments, and details are not repeated here.

It is easy for those skilled in the art to understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (10)

1. A head pose estimation method, characterized in that, comprises the steps: Identify images that contain human faces; The image is input to the pre-trained layered prediction network, and the pitch angle, yaw angle and roll angle of the attitude of the face are predicted to estimate the head pose of the face; the layered prediction network includes: a backbone network , a feature pyramid network, a dimensionality reduction module, and a hierarchical prediction module; the backbone network is used to extract image space features of different sizes, and the feature pyramid network is used to fuse image space features of different sizes to obtain fusion features, and the reduction The dimension module is used to perform dimensionality reduction in three different dimensions on the fusion feature to obtain spatial features of the image in three dimensions, and different dimensions correspond to different image channel numbers; the layered prediction module includes: three fully connected layers; The three fully connected layers predict the spatial features of the three dimensions respectively, and each fully connected layer predicts an angle of the orientation of the face pose, so that the layered prediction network predicts the orientation of the face pose toward the three dimensions. Each angle focuses on different image areas, reducing the mutual interference between the three angle predictions; the size is in pixels. 2. The method according to claim 1, wherein the loss function in the training process of the layered prediction network adopts a self-adjusting loss limitation coefficient, so that the average absolute error of the three fully connected layer prediction angles is less than the threshold When the cross entropy loss item of the prediction angle is greater than the mean square error loss item of the prediction angle, the loss size reversal problem is corrected, and when the average absolute error of the prediction angle of the three fully connected layers is not less than the threshold value, the above two The error penalty brought by the loss item makes the hierarchical prediction network converge at a faster speed during the training process; when the average absolute error of the three fully connected layer prediction angles is less than the threshold, the error penalty of the hierarchical prediction network is The first penalty, when the average absolute error of the prediction angles of the three fully connected layers is not less than the threshold, the error penalty of the layered prediction network is the second penalty, and the correction loss size reverse problem means that the first penalty is controlled to be less than the second penalty, and the score is guaranteed The layer prediction network can train and learn normally. 3. The method according to claim 1, wherein the backbone network includes four residual blocks; the face image is sequentially processed through the four residual blocks, and four kinds of image space features with decreasing sizes are sequentially obtained ; The above four size-decreasing image space features are fused by the feature pyramid network. The fusion strategy is to first fuse the first size space feature with the second size space feature, and combine the first size space feature with the third size space feature. The spatial features are fused to obtain the new second-dimensional spatial features and the new third-dimensional spatial features, and then the new second-dimensional spatial features are fused with the new third-dimensional spatial features, and the new third-dimensional spatial features are fused. The second dimension space feature is fused with the fourth dimension space feature to obtain the updated third dimension space feature and the new fourth dimension space feature, and finally the updated third dimension space feature and the new dimension space feature The fourth size space feature is fused to obtain the fusion feature of the fourth size; wherein, the size from the first size to the fourth size decreases step by step; The dimensionality reduction module includes three convolution kernels; the fusion features of the fourth size are sequentially processed by the three convolution kernels, and each convolution kernel performs a dimensionality reduction on the input image features, and successively obtains different sizes. The spatial characteristics of the three dimensions change and the number of channels decreases step by step. 4. The method according to claim 3, characterized in that, the adjustment formulas of the three angles predicted by the layered prediction network are:

in,

as well as

respectively represent the predicted values of pitch angle, yaw angle and roll angle; K ₁ , K ₂ and K ₃ are the weight factors of the three convolution kernels respectively; Γ ₁ , Γ ₂ and Γ ₃ are the dimensionality reduction of the fusion features The three-dimensional spatial features obtained by the three convolution kernels of the module; The relationship between Γ ₁ , Γ ₂ , and Γ ₃ satisfies the following formula:

Among them, W ₁ is the feedback parameter from the first convolution kernel to the second convolution kernel of the dimensionality reduction module, W ₂ is the feedback parameter from the second convolution kernel to the third convolution kernel of the dimensionality reduction module, b ₄ is the new bias term brought by the dimensionality reduction from the first convolution kernel to the second convolution kernel, and b ₅ is the new bias term brought by the dimensionality reduction from the second convolution kernel to the third convolution kernel. 5. The method according to any one of claims 1 to 4, wherein the loss function of the hierarchical prediction network

for:

in,

is the value predicted by the hierarchical prediction network for the head pose, y is the real value of the head pose of the face in the image, and β is the loss restriction coefficient, which is constructed by the relationship between the mean square error loss and the cross entropy loss , k is the number of angle categories, σ is the sigmod function, L _mse represents the mean square error loss, Y _ic represents the one-hot code formed according to the angle category,

is the category to which the predicted angle belongs. 6. A head posture estimation system, characterized in that, comprising: A human face image determining unit, configured to determine an image containing a human face; The head posture estimation unit is used to input the image into the pre-trained layered prediction network, and predicts the pitch angle, yaw angle and roll angle towards the face posture to estimate the head posture of the human face; The layered prediction network includes: a backbone network, a feature pyramid network, a dimensionality reduction module, and a layered prediction module; the backbone network is used to extract image space features of different sizes, and the feature pyramid network is used to fuse image space features of different sizes , to obtain the fusion feature, the dimensionality reduction module is used to perform dimensionality reduction in three different dimensions on the fusion feature, to obtain the spatial features of the image in three dimensions, and different dimensions correspond to different image channel numbers; the layered prediction module Including: three fully connected layers; the three fully connected layers respectively predict the spatial features of the three dimensions, and each fully connected layer predicts an angle towards the orientation of the face posture, so that the layered prediction The network predicts the face poses of the three angles to focus on different image areas, reducing the mutual interference between the three angle predictions; the size is in pixels. 7. The system according to claim 6, wherein the loss function in the layered prediction network training process used by the head pose estimation unit adopts a self-adjusting loss restriction coefficient to predict When the average absolute error of the angle is less than the threshold, correct the loss size reversal problem caused by the cross-entropy loss item of the predicted angle being greater than the mean square error loss item of the predicted angle, and the average absolute error of the predicted angle in the three fully connected layers is not less than the threshold When , the error penalty brought by the above two loss items is increased, so that the layered prediction network converges at a faster speed during the training process; when the average absolute error of the prediction angle of the three fully connected layers is less than the threshold, the divided The error penalty of the layer prediction network is the first penalty. When the average absolute error of the prediction angle of the three fully connected layers is not less than the threshold, the error penalty of the layered prediction network is the second penalty, and the correction loss size reverse problem refers to controlling the first penalty. If it is less than the second penalty, it ensures that the hierarchical prediction network can be trained and learned normally. 8. The system according to claim 6, wherein the backbone network used by the head pose estimation unit includes four residual blocks; the face image is sequentially processed through the four residual blocks to obtain four sequentially. Image space features with decreasing size; the above four image space features with decreasing size are fused by the feature pyramid network. The dimension space feature is fused with the third dimension space feature to obtain the new second dimension space feature and the new third dimension space feature, and then the new second dimension space feature and the new third dimension space feature Features are fused, and the new second dimension space feature is fused with the fourth dimension space feature to obtain the updated third dimension space feature and the new fourth dimension space feature, and finally the updated No. The three-dimensional spatial features and the new fourth-dimensional spatial features are fused to obtain the fusion features of the fourth size; wherein, the sizes from the first size to the fourth size are gradually reduced; the dimensionality reduction module includes Three convolution kernels; the fusion features of the fourth size are sequentially processed by the three convolution kernels, and each convolution kernel performs a dimensionality reduction on the input image features, and sequentially obtains the same size and the number of channels step by step Decreasing spatial characteristics of the three dimensions. 9. The system according to claim 8, wherein the adjustment formula of the three angles predicted by the layered prediction network used by the head pose estimation unit is:

in,

as well as

respectively represent the predicted values of pitch angle, yaw angle and roll angle; K ₁ , K ₂ and K ₃ are the weight factors of the three convolution kernels respectively; Γ ₁ , Γ ₂ and Γ ₃ are the dimensionality reduction of the fusion features The three-dimensional spatial features obtained by the three convolution kernels of the module; the relationship between Γ ₁ , Γ ₂ , and Γ ₃ satisfies the following formula:

Among them, W ₁ is the feedback parameter from the first convolution kernel to the second convolution kernel of the dimensionality reduction module, W ₂ is the feedback parameter from the second convolution kernel to the third convolution kernel of the dimensionality reduction module, b ₄ is the new bias term brought by the dimensionality reduction from the first convolution kernel to the second convolution kernel, and b ₅ is the new bias term brought by the dimensionality reduction from the second convolution kernel to the third convolution kernel. 10. The system according to any one of claims 6 to 9, wherein the loss function of the hierarchical prediction network used by the head pose estimation unit

for:

in,

is the value predicted by the hierarchical prediction network for the head pose, y is the real value of the head pose of the face in the image, and β is the loss restriction coefficient, which is constructed by the relationship between the mean square error loss and the cross entropy loss , k is the number of angle categories, σ is the sigmod function, L _mse represents the mean square error loss, Y _ic represents the one-hot code formed according to the angle category,

is the category to which the predicted angle belongs.

Images