CN112308957B - An automatic generation method of optimal fat and thin face portrait images based on deep learning - Google Patents

Abstract

The invention discloses a method for automatically generating an optimal fat and thin portrait image based on deep learning. The face-related parameters include face pose parameters; the face portrait image is input into the trained best fat and thin estimation model based on deep learning, and the best fat and thin scale of the face is output; The best fat and thin scale is the input. According to the 3D face fat and thin adjustment algorithm, the 3D face model is adjusted to generate the best fat and thin 3D face model; The 3D model is projected on the 2D plane to get the best fat and thin face; through the front and back background fusion algorithm, the best fat and thin face is seamlessly embedded into the face portrait image to get the best fat and thin face portrait image. The present invention can be used in social, animation, game and other fields.

Description

An automatic generation method of optimal fat and thin face portrait images based on deep learning

technical field

The invention relates to the technical field of portrait editing, in particular to a method for automatically generating portrait images of optimal fat and thin faces based on deep learning.

Background technique

The human face plays a vital role in expressing personal characteristics and creating a first impression. Most people want all their photos on Facebook or Instagram to look good. So portrait editing plays a vital role in many applications such as social media, advertising, visual effects, fitness incentives, etc. To this end, a series of face portrait editing methods have been proposed to meet the needs of various aspects of production and life.

The facial attractiveness of a portrait image is affected by two main factors, facial texture and facial shape.

Facial texture is determined solely by the color on the face and can be improved with direct color adjustments such as brightness enhancement. For example, Shih et al. transfer the style of one portrait to another. Works such as face makeup transfer, face light transfer, and face relighting generate convincing results on textures, but the lack of shape editing limits application possibilities.

Therefore, many works related to face shape editing have been proposed. Shih et al. solved the problem of face deformation under wide-angle lens by correcting camera distortion. Similarly, methods such as personalized exaggerated face generation and face personalized analysis and extraction have also been proposed to solve problems related to face shape editing.

To improve the attractiveness of human faces, Leyvand et al. (Data-Driven Enhancement of Facial Attractiveness. ACM Transactions on Graph-ics27, 3, Article 38 (Aug. 2008), 9 pages.) proposed an aesthetic-inspired face shape editing method , according to public aesthetic standards, the nose, mouth, eyes, eyebrows, face shape, etc. are adjusted on the front face of the human face by relying on high-precision facial feature points in 2D. However, this method cannot be used to beautify the face. Feature point accuracy is extremely sensitive.

Liao et al. (Enhancing the Symmetry and Proportion of 3D FaceGeometry. IEEE Transactions on Visualization and ComputerGraphics 18, 10 (2012), 1704-1716.) extended the beautification standard to 3D human faces, and added side face beautification adjustment content, But their method focuses on the adjustment of salient structures (facial features and silhouette proportions) rather than considering the underlying physiological structure of the human face, so it is prone to unreasonable results.

Zhao et al. (Parametric Reshaping of Portrait Images for Weight-Change. IEEE Computer Graphics and Applications 38, 1 (2018), 77-90.) proposed a soft tissue thickness regression model based on sparse feature points on the face region based on a single parameter fat and thin adjustment method. This method adjusts the face based on real face soft tissue depth data. Since it only relies on sparse face feature points (rather than full face information) to reconstruct and deform 3D faces, it will produce unreasonable faces (especially is when dealing with exaggerated poses and expressions). Furthermore their method is not automatic, thus requiring the user to manually adjust the parameters to obtain satisfactory results.

A morphable model for the synthesis of 3D faces. Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques-SIGGRAPH 1999, 187-194 discloses a method to characterize 3D faces using a three-dimensional morphable model (3DMM) based parametric representation, and then automatically Estimate the face fat and thin scale, and finally adjust the face reasonably to obtain the best fat and thin results.

However, due to the representation limitation of 3DMM, maintaining face ID while adjusting for fat and thin is still a challenging problem. Moreover, automatic face fat and thin estimation requires labeled data of different groups of people such as gender and age. In order to ensure the accuracy and consistency of annotations, it is best to have labeled data of the same person with different fat and thin scales. However, considering that the face fatness and thinness of an individual usually remains constant for a long time, it is difficult to obtain different fatness and thinness data of the same person.

SUMMARY OF THE INVENTION

In order to solve the problem of how to quickly and automatically generate the best fat and thin face, especially the problem of the rationality of fat and thin adjustment, the present invention provides an automatic generation method of the best fat and thin portrait image based on deep learning. Expression face portrait images, this method can automatically generate the best fat and thin face portraits that are stylish, attractive and reasonable.

In order to solve the above-mentioned technical problems, the present invention provides the following technical solutions:

A method for automatic generation of optimal fat and thin face portrait images based on deep learning, including the following steps:

S1, generate a three-dimensional face model, a face-related parameter and a texture map of a two-dimensional face in a face portrait image, and the face-related parameter includes a face pose parameter;

S2. Input the face portrait image into the trained best fat and thin estimation model based on deep learning, output the best fat and thin scale of the face, and the best fat and thin estimation model of the human face contains The face database of public face aesthetic fat and thin annotation is obtained by training;

S3. Take the output optimal fatness and thinness scale as input, adjust the 3D face model according to the 3D face fatness and thinness adjustment algorithm, and generate the best fatness and thinness 3D face model;

S4. Project the textured best fat and thin three-dimensional model on a two-dimensional plane according to the face posture parameters to obtain the best fat and thin face; the textured best fat and thin three-dimensional face model is based on texture mapping;

S5. Through the front and back background fusion algorithm, the best fat and thin face is seamlessly embedded in the face portrait image, and the best fat and thin face portrait image is obtained.

Compared with the prior art, the present invention has the following beneficial effects:

(1) The present invention provides a simple, effective and stable method for automatically generating the optimal fatness and thinness of a portrait image. The invention can process portrait images irrelevant to expressions and postures, and automatically generate reasonable, high-fidelity, stylish, and identity-preserving human face portraits. It has high application value in actual social media and the Internet.

(2) The three-dimensional face fatness and thinness adjustment algorithm provided by the present invention can balance the dense information constraints and rationality constraints on 3DMM, face identity features, face fatness and thinness, etc., to generate a convincing symmetrical face effect. It is suitable for applications that require free control of face fat and thin.

(3) The face fatness and thinness database provided by the present invention includes public aesthetic fatness and thinness degree labels, which can be used to train a fatness and thinness estimation model based on deep learning, and is suitable for applications such as automatic portrait editing.

Description of drawings

FIG. 1 is a flow chart of a method for automatically generating an optimal fat and thin face portrait image based on deep learning provided by the present invention.

Detailed ways

As shown in Figure 1, the method for automatically generating the best fat and thin face portrait image based on deep learning provided by the present invention includes the following steps:

S1. Generate a three-dimensional face model, face-related parameters, and texture mapping of a two-dimensional face in a face portrait image, where the face-related parameters include face pose parameters.

S11, input the two-dimensional face detected in the face portrait image into the face feature point detection algorithm, and locate the face key points of the two-dimensional face;

S12, rebuilding a three-dimensional face model according to a monocular vision three-dimensional face reconstruction algorithm and calculating a face-related parameter, where the face-related parameter includes a face identity feature parameter, a face expression parameter and a face pose parameter;

S13. Obtain a texture map according to the face portrait image and the three-dimensional face model.

Texture mapping is a method for defining high frequency detail, surface texture or color information on computer-generated graphics or 3D models.

The texture mapping can map pixels from a face portrait picture to a three-dimensional face model, and "wrap" the face portrait picture around the three-dimensional face model.

The three-dimensional face model is reconstructed according to the monocular vision three-dimensional face reconstruction algorithm and the face-related parameters are calculated, as follows:

所述的三维人脸模型的通过平均人脸

与一组人脸形状偏移量的线性组合来表示：S121, as shown in formula (1), characterize the three-dimensional face model as a parameter expression based on 3DMM, and the three-dimensional face model is expressed as a fixed topology grid composed of n vertices

The three-dimensional face model through the average face

It is represented by a linear combination with a set of face shape offsets:

是人脸身份特征形状偏移量，

是人脸表情特征形状偏移量，算法包含N_id个形状参数，N_exp个表情参数；α代表人脸身份特征参数，β代表人脸表情参数，

代表3n维空间；in,

is the face identity feature shape offset,

is the facial expression feature shape offset. The algorithm includes N _id shape parameters and N _exp expression parameters; α represents the facial identity feature parameter, β represents the facial expression parameter,

Represents a 3n-dimensional space;

S122, obtain face-related parameters according to the optimal solution of the energy equation described in formula (2);

E(P)=w _f E _f (P)+w _r E _r (P), Equation (2)

Among them, P represents the relevant parameters of the face, P={R, t, α, β}, R, t represent the rotation and translation of the face pose parameters, respectively, E _f (P) represents the three-dimensional face through the key points of the face. The energy term fitted to the two-dimensional face, E _r (P) is the regularization term, and w _f and w _r are the coefficients of the two energy terms, respectively.

The optimization problem of the energy equation is a convex optimization problem, which can be transformed into a least squares problem to obtain the optimal solution.

For the monocular vision 3D face reconstruction algorithm, please refer to Patrik Huber, Guosheng Hu, RafaelTena, Pouria Mortazavian, P Koppen, William J Christmas, Matthias Ratsch, and Josef Kittler.2016.A Multiresolution 3D Morphable Face Model and FittingFramework.INSTICC, 79-86.

and Christian Theobalt.2018.Deep Video Portraits.ACM TOG 37，4，Article 163(2018)，14pages。Energy equation see Hyeongwoo Kim, Pablo Carrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael

and Christian Theobalt. 2018. Deep Video Portraits. ACM TOG 37, 4, Article 163 (2018), 14pages.

S2. Input the face portrait image into the trained best fat and thin estimation model based on deep learning, output the best fat and thin scale of the face, and the best fat and thin estimation model of the human face contains The public face aesthetics are obtained by training the face database with fat and thin annotations.

S21. Construct a face database including public face aesthetic fat and thin annotations;

S211, using the method in S1 to generate a three-dimensional face model, face-related parameters and texture mapping of the two-dimensional face in the original face portrait photo of the face database;

S212. Using the method in S3, set a plurality of different fat and thin scales to replace the optimal fat and thin scale as input, and obtain corresponding adjusted three-dimensional face models respectively. The value range of the fat and thin scale is {-2.0, - 1.6, -1.2, -0.8, -0.4, 0.0, 0.4, 0.8};

S213. Using the methods in S4 to S5, project the adjusted three-dimensional face model onto a two-dimensional plane in turn, and then embed it into the corresponding original face portrait photos to obtain a sequence of face portrait photos with different fatness and thinness;

S214. Ask multiple raters to select the best fat and thin face portrait photo from the face portrait photo sequence, and use the corresponding fat and thin scale as the fat and thin baseline u of the face, and set the public profile of other face portrait photos. The face aesthetic fatness and thinness scale s ^* = vu, where v is the fatness and thinness scale corresponding to other face portrait photos;

S215: Perform S221-S224 processing on each original face portrait photo in the face database, so as to obtain a face database containing public face aesthetic fatness and thinness annotations.

S22, using the face database containing the public face aesthetic fat and thin labels to train the best face fat and thin estimation model;

The described best face fat and thin estimation model is based ^on the network structure of ResNet, including feature extraction layer and classifier ^. Among them, s is the fat and thin scale estimated by the best face fat and thin estimation model; s ^* represents the public face aesthetic fat and thin scale.

See He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778., Fixed features Extraction layer, modify the classifier to 1 class.

S23, input the face portrait image into the best face fat and thin estimation model to obtain the best fat and thin scale.

The construction method of the face database including the public face aesthetic fatness and thinness annotation according to the present invention uses a single parameter (the public face aesthetic fatness and thinness scale) to describe the face fatness and thinness, and avoids a wide variety of face parameters (such as forehead, parameter adjustment of cheeks, chin, etc.).

S3. Taking the output optimal fatness and thinness scale as the input, adjust the three-dimensional face model according to the three-dimensional face fatness and thinness adjustment algorithm, and generate the best fatness and thinness three-dimensional face model.

S31. First, fix the expression parameters in the three-dimensional face model, simplify the three-dimensional face model as X(α), and express any point in X(α) as shown in the following formula:

是P_id对应于x_i的子矩阵；in,

is the submatrix of P _id corresponding to x _i ;

S32, calculating the optimal solution of formula (5) to obtain the new position Y of each vertex on the three-dimensional face model;

E(Y,α)=E _shape (Y)+E _3DMM (Y,α)+E _ID (Y), formula (5)

是三维人脸模型中n个顶点的新位置；E_shape(Y)代表稀疏胖瘦模型约束，E_3DMM(Y，α)代表人脸合理性约束，E_ID(Y)代表人脸身份特征约束，

代表3维空间；in,

is the new position of n vertices in the 3D face model; E _shape (Y) represents the sparse fat and thin model constraint, E _3DMM (Y, α) represents the face rationality constraint, and E _ID (Y) represents the face identity feature constraint ,

Represents a 3-dimensional space;

The formula for calculating E _shape (Y) is as follows:

E _shape (Y)=∑ _j (y _I(j) -x′ _I(j) ) ² Equation (6)

Among them, I(j) is the index of the jth face feature point, y _I(j) represents the new position of the jth face feature point; x′ _I(j) represents the jth face feature point according to the best fat The new position after thin scale adjustment, the calculation formula is as follows:

x′ _I(j) = x _I(j) (α ₀ )+δBMI·b _j ·n _I(j) Formula (4)

是第j个人脸特征点的法向，b表示所有特征点的回归系数，b＝{b_j}，j＝1，2，...，52；Among them, x _I(j) (α ₀ ) represents the original position of the jth face feature point, BMI represents the height-to-weight ratio difference, and δBMI represents the optimal fat and thin scale,

is the normal direction of the jth face feature point, b represents the regression coefficient of all feature points, b={b _j }, j=1, 2,..., 52;

The sparse fat-thin model constraint E _shape (Y) is based on forensic research (see Sven De Greef, Peter Claes, Dirk. Vandermeulen, Wouter Mollemans, Paul Suetens, and Guy Willems. 2006. Large-Scale In-Vivo Caucasian Facial Soft Tissue Thick- ness Database for Craniofacial Reconstruction. Forensic science international 159 (2006), S126-S146.) The sparse face fat and thin adjustment model is used for face fat and thin constraints. The sparse face fat and thin model uses a linear regression model to represent the mapping relationship between the soft tissue depth and age and height-to-weight ratio (BMI) of 52 three-dimensional face feature points.

E _3DMM (Y, α) constrains faces within a reasonable face space. The calculation formula of E _3DMM (Y, α) is as follows:

是人脸身份特征参数α的下则项；where ∈ is a diagonal matrix consisting of the eigenvectors of P _id ,

is the following term of the face identity feature parameter α;

E _ID (Y) constrains the face before and after adjustment of fat and thin to be the same face. The formula for calculating E _ID (Y) is as follows:

Among them, Δ( ) is the Laplacian operator. Since the E _shape does not modify the facial features, the operator is sufficient to maintain the facial features.

E(Y, α) constitutes a convex optimization problem, and the new position Y of all vertices after fat and thin adjustment can be obtained by solving the least squares problem.

S33 , adding the expression P _exp β ₀ to the optimal solution Y to obtain the optimal fat and thin three-dimensional face model.

The three-dimensional face adjustment algorithm described in the present invention refers to a three-dimensional face adjustment algorithm that can balance the identity features of the face, the density constraints and rationality constraints of 3DMM, and the fat and thin constraints of the sparse face adjustment model to generate a convincing symmetrical face effect. Face fat and thin adjustment algorithm.

S4. Project the textured best fat and thin three-dimensional model onto a two-dimensional plane according to the face pose parameters to obtain the best fat and thin face, the textured best fat and thin two-dimensional face model Obtained from texture mapping.

The projection described is an orthogonal projection.

S5. Through the front and back background fusion algorithm, the best fat and thin face is seamlessly embedded in the face portrait image, and the best fat and thin face portrait image is obtained.

S51, using texture mapping to establish the correspondence between the best fat and thin face to the face portrait image;

S52 , using the background grid deformation of the corresponding guide face portrait picture, so that the best fat and thin face is seamlessly embedded in the face portrait picture after the background grid deformation, so as to obtain the best fat and thin face portrait image.

See YiChang Shih, Wei-Sheng Lai, and Chia-KaiLiang. 2019. Distortion-Free Wide-Angle Portraits on Camera Phones. ACM TOG 38, 4, Article 61, 12 pages for the described front and back background fusion algorithms.

Claims (4)

1. An optimal fat and thin face portrait image automatic generation method based on deep learning is characterized by comprising the following steps:

s1, generating a three-dimensional face model, face related parameters and texture mapping of a two-dimensional face in the face portrait image, which are as follows:

s11, inputting the two-dimensional face detected in the face portrait image into a face feature point detection algorithm, and positioning face key points of the two-dimensional face;

s12, reconstructing a three-dimensional face model according to a monocular vision three-dimensional face reconstruction algorithm and calculating face related parameters, wherein the face related parameters comprise face identity characteristic parameters, face expression parameters and face posture parameters, and the method specifically comprises the following steps:

s121, as shown in formula (1), representing a three-dimensional face model as a parameter expression based on 3DMM, wherein the three-dimensional face model is represented as a fixed-topology grid formed by n vertexes

The three-dimensional face model is obtained by averaging the faces

Expressed in linear combination with a set of face shape offsets:

wherein,

is the offset of the shape of the face identity feature,

is the offset of the facial expression characteristic shape, and the algorithm comprises N_idA shape parameter, N_expAn individual expression parameter; alpha represents the human face identity characteristic parameter, beta represents the human face expression parameter,

representing a 3 n-dimensional space;

s122, obtaining face related parameters according to the optimal solution of the energy equation of the formula (2);

E(P)＝w_fE_f(P)+w_rE_r(P)in the formula (2)

Wherein, P represents face related parameters, P ═ { R, t, α, β }, R, t represent rotation and translation of face pose parameters, respectively, E_f(P) represents an energy term fitting a three-dimensional face to a two-dimensional face through face key points, E_r(P) is a regularization term, w_f，w_rCoefficients of two energy terms, respectively;

s13, obtaining texture mapping according to the face portrait image and the three-dimensional face model;

s2, inputting the face portrait image into a trained optimal face fat-thin estimation model based on deep learning, and outputting an optimal face fat-thin scale, wherein the optimal face fat-thin estimation model is obtained by training a face database containing public face aesthetic fat-thin labels, and the method specifically comprises the following steps:

s21, constructing a face database containing the aesthetic fat and thin annotation of the public face, which comprises the following specific steps:

s211, generating a three-dimensional face model, face related parameters and texture mapping of a two-dimensional face in the original face portrait photo of the face database by adopting the method in S1;

s212, setting a plurality of different fat-thin scales to replace the optimal fat-thin scale as input by adopting the method in S3, and respectively obtaining the correspondingly adjusted three-dimensional face model, wherein the value range of the fat-thin scales is { -2.0, -1.6, -1.2, -0.8, -0.4, 0.0, 0.4, 0.8 };

s213, projecting the adjusted three-dimensional face model onto a two-dimensional plane in sequence by adopting the method in S4-S5, and then embedding the three-dimensional face model into corresponding original face portrait photos to obtain face portrait photo sequences with different fat and thin sizes;

s214, a plurality of scoring personnel are requested to select the face portrait photo with the best fat and thin from the face portrait photo sequence, the corresponding fat and thin scale is used as the fat and thin base line u of the face, and the public face aesthetic fat and thin scale S of other face portrait photos is set^*V-u, wherein v is the fat-thin scale corresponding to other face portrait photos;

s215, processing S211-S214 is carried out on each original face portrait photo in the face database, and therefore the face database containing aesthetic fat and thin labels of public faces is obtained;

s22, training an optimal face fat-thin estimation model by using a face database containing public face aesthetic fat-thin labels;

the optimal human face fat-thin estimation model is based on a network structure of ResNet and comprises a feature extraction layer and a classifier, wherein the classifier is of type 1; in the training process, the sigma is s-s^*||²Minimizing to a target, wherein s is a fat-thin scale estimated by the optimal fat-thin estimation model of the human face; s^*Representing aesthetic fat and thin dimensions of public human faces;

s23, inputting the face portrait image into the face optimal fat-thin estimation model to obtain an optimal fat-thin scale;

s3, adjusting the three-dimensional face model by taking the output optimal fat-thin scale as input according to a three-dimensional face fat-thin adjustment algorithm, and generating the optimal fat-thin three-dimensional face model as follows:

s31, firstly, fixing expression parameters in the three-dimensional face model, simplifying the three-dimensional face model to be X (alpha), and expressing any point in the X (alpha) as shown in the following formula:

wherein, alpha represents the face identity characteristic parameter;

is P_idCorresponds to x_iA sub-matrix of (a);

s32, calculating the optimal solution of the formula (5) to obtain the new position Y of each vertex on the three-dimensional face model;

E(Y，α)＝E_shape(Y)+E_3DMM(Y，α)+E_ID(Y), formula (5)

Wherein,

is the new position of n vertexes in the three-dimensional face model; e_shape(Y) represents sparse fat-lean model constraints, E_3DMM(Y, α) represents a face reasonableness constraint, E_ID(Y) represents a face identity feature constraint,

representing a 3-dimensional space;

E_shapethe calculation formula of (Y) is as follows:

E_shape(Y)＝∑_j(y_I(j)-x′_I(j))²formula (6)

Wherein I (j) is the index of the jth individual face feature point, y_I(j)A new location representing a jth personal face feature point; x'_I(j)The new position of the j-th individual face feature point adjusted according to the optimal fat-thin scale is represented by the following calculation formula:

x′_I(j)＝x_I(j)(α₀)+δBMI·b_j·n_I(j)formula (4)

x_I(j)(α₀) Representing the original location of the jth individual's face feature point, BMI representing height-to-weight ratio difference, δ BMI representing the optimal fat-thin scale,

is the normal to the jth face feature point, b represents the regression coefficients for all feature points, b ═ b_jJ is an integer of 1-52;

E_3DMMthe calculation formula of (Y, α) is as follows:

wherein e is represented by P_idIs used to form a diagonal matrix of feature vectors,

is a regular term of the face identity characteristic parameter alpha;

E_IDthe calculation formula of (Y) is as follows:

where Δ () is the laplacian operator;

s33 expression P_expAdding beta into the optimal solution Y to obtain an optimal fat and thin three-dimensional face model;

s4, projecting the optimal fat-thin three-dimensional model with textures onto a two-dimensional plane according to the human face posture parameters to obtain an optimal fat-thin human face; the optimal fat and thin three-dimensional face model with the texture is obtained according to texture mapping;

and S5, seamlessly embedding the optimal fat and thin face into the face portrait image through a front and back background fusion algorithm to obtain the optimal fat and thin face portrait image.

2. The method for automatically generating an optimal fat-thin human face portrait image based on deep learning of claim 1, wherein the projection of the step S4 is an orthogonal projection.

3. The method for automatically generating an optimal fat-thin face portrait image based on deep learning of claim 1, wherein the optimal fat-thin face is seamlessly embedded into the face portrait image through a front-back background fusion algorithm to obtain the optimal fat-thin face portrait image, which is specifically as follows:

s51, establishing the correspondence from the face with the best fat and thin to the face portrait image by utilizing texture mapping;

and S52, utilizing the corresponding background grid deformation for guiding the human face portrait picture to enable the optimal fat and thin human face to be seamlessly embedded into the human face portrait picture after the background grid deformation, and obtaining the optimal fat and thin human face portrait picture.

4. An apparatus for automatically generating an optimal fat-thin face portrait image based on deep learning, comprising a computer memory, a computer processor and a computer program stored in the computer memory and executable on the computer processor, wherein the computer processor executes the method for automatically generating an optimal fat-thin face portrait image based on deep learning according to any one of claims 1 to 3.

Images