CN115457171A - Efficient expression migration method adopting base expression space transformation - Google Patents

Abstract

The invention provides a high-efficiency expression migration method adopting base expression space transformation, which comprises the following steps: a preprocessing stage and a facial expression migration stage; the pre-processing stage comprises: acquiring a source model O, wherein the source model O is a basic expression model and comprises expression grids under various different expressions; performing facial expression reconstruction on the source model O and the target model T to obtain expression description parameters of the source model O and the target model T; similarity estimation is carried out according to the expression description parameters to obtain an expression parameter conversion matrix and a similarity matrix; the facial expression migration stage comprises: and inputting the role picture into the base expression model, and calculating to obtain expression description parameters of the target expression model according to the expression description parameters in the base expression model and the similarity matrix to finish expression migration.

Description

An efficient expression transfer method using base expression space transformation

technical field

The invention relates to computer vision, in particular to an efficient expression transfer method using basic expression space transformation.

Background technique

The traditional method transfers the facial expression from the actor in the source video to the actor in the target video in real time, so as to achieve temporary control of the facial expression of the target actor, such as paper 1: "Real-time expression transfer forfacial reenactment" often deforms the face and detail transfer and photorealistic re-rendering into the target video, making the newly synthesized expressions almost indistinguishable from the real video. To achieve this, a commercial RGB-D sensor is used to accurately capture the facial representations of source and target subjects in real time. For each frame, parametric models for logotype, expression, and skin albedo are combined with input color and depth data, and scene lighting is reconstructed. For expression transfer, the difference between the source and target expressions in parameter space is computed, and the target parameters are modified to match the source expression. A major challenge is to convincingly re-render the synthesized target surface into the corresponding video stream. This requires careful consideration of lighting and shading design, both of which must correspond to real-world environments. The authors demonstrate methods in a real-time setting that modify a video conference feed to match the facial expressions of different people (e.g. translators) in real time.

The disadvantage of this method is that the method calculates the difference between two expressions in the parameter space, and then modifies the expression parameters of the target to match the source expression. Parameterized by spherical harmonics, these can cause artifacts in general environments (e.g., with strong subsurface scattering, high-frequency lighting variations, or self-shadowing). Second, our approach's tracker uses dense depth and color information, which allows tight fitting but also leads to large residuals.

Paper 2: "Face2Face: Real-time Face Capture and Reenactment of RGBVideos" proposes a method for replaying monocular target video sequences in real time (for example, Youtube videos). The source sequence is also a monocular video stream, captured in real time using a commodity webcam. The goal of this paper is to animate the facial expressions of a target video by a source actor and re-render the manipulated output video in a photorealistic manner. To this end, the insufficient problem of facial identity recovery from monocular videos is first addressed by non-rigid model-based binding. At runtime, the facial expressions of the source and target videos are tracked using dense photometric coherence measurements. Replay is then achieved by transferring deformations quickly and efficiently between source and target. The closest match to the retargeted expressed buccal interior was retrieved from the target sequence and warped to produce an accurate fit. Finally, the composited target surface is convincingly re-rendered on top of the corresponding video stream so that it blends seamlessly with real-world lighting.

This method suffers from the following disadvantages: The method is challenging for scenes where the face is hidden by long hair and beard. Furthermore, the method only reconstructs and tracks a low-dimensional curved shape model (76 expression coefficients), which ignores fine-scale static and transient surface details, in a sequence that is too short, or when the target remains stationary , we cannot learn specific oral behaviors. In this case, temporal aliasing can be observed due to the fact that the object space of the retrieved mouth samples is too sparse.

In the process of expression transfer, the core issue is how to establish a mapping for two sets of expressions, that is, to evaluate the similarity of expressions. Most of the current similarity estimation methods are determined by the spatial position and attitude of the vertices or triangular faces of the mesh model, which requires finding the corresponding vertices or triangular faces of the model. At present, the methods for similarity estimation mainly include methods based on feature points in the image, based on model vertices, and based on triangular faces. Methods based on feature points or model vertices basically use the offset direction and size of comparison points for similarity estimation. Based on the triangular face, the similarity expression is generated by finding the corresponding triangular face of the model and transforming the direction of the corresponding triangular face. However, this method does not work well when the shape of the target face and the source face are relatively different.

Therefore, how to realize a high-efficiency, ultra-realistic, and universally applicable facial expression transfer method is an urgent problem to be solved in the present invention.

Contents of the invention

In view of this, the present invention provides a kind of efficient expression transfer method that adopts base expression space transformation, comprises: preprocessing stage and facial expression transfer stage; Said preprocessing stage includes:

Obtaining the source model O, the source model O is a base expression model, which includes expression grids under multiple different expressions;

Perform facial expression reconstruction on the source model O and the target model T to obtain the expression description parameters of the two; perform similarity estimation according to the expression description parameters to obtain the expression parameter conversion matrix and similarity matrix;

The facial expression transfer stage includes: inputting character pictures into the base expression model, calculating the expression description parameters of the target expression model according to the expression description parameters in the base expression model and the similarity matrix, and completing the expression transfer.

In particular, the facial expression reconstruction includes: defining a set of facial feature points, finding the corresponding vertex serial number in the 3D model; for any input facial expression photo, detecting the two-dimensional coordinates of the defined feature points, An optimization equation is established so that the position of the feature point in the model projected onto the plane is the smallest distance from the position of the 2D point in the image; wherein the optimization equation is expressed as follows:

E＝∑||P(x _v )-p _v || ₂

Wherein P is a projection matrix, xv is the three-dimensional position of the _vth facial feature point, which can be obtained from the optimization equation, and pv is the projection coordinate of the corresponding _vth feature point on a two-dimensional plane.

In particular, the source model O selects an expressionless grid model as a neutral reference, and the offsets of other expressions relative to the neutral reference form the superposition effect of different expressions, and the offsets constitute the expression descriptions of different facial expressions parameter.

In particular, for the _i -th patch, under different expressions, the offset of the feature point relative to the neutral expression is δi, which is used as the description of the current expression, then the expression gap between the source model O and the target model T is expressed by the following formula:

Among them, E _Ci constrains the direction of feature point offset between source model O and target model T; E _Di constrains the distance of feature point offset between source model O and target model T;

δ _Oi is the expression description of the i-th patch of the source model O _, δ _Ti is the expression description of the i-th patch of the model T; The offset of v vertices, V _i is all vertices of the i-th patch.

In particular, the optimization equation for expression similarity estimation for each patch is as follows:

E _Si ＝λ _M *E _Di *E _Ci +λ _D *E _Di +λ _C *E _Ci

Where E _Ci is the expression gap between the source model O and the target model T, E _Di is a reference shape added as the second constraint to obtain accurate facial similarity estimation, and λ _M , λ _D , and λ _C are weight values.

By solving the above equations for all the faces, for each expression in the source model O, a corresponding similar expression of a target model T can be obtained, and all similar expressions can form a set of expression parameter transformation matrix equivalent to mixed deformation , the expression parameter transformation matrix of the equivalent blend shape corresponds to the expression parameters of the target model T. For the blend shape parameters of each patch, a similarity matrix S is formed. Through this similarity matrix, any source model can be The expression parameter α _O of O is transformed into the expression parameter α _T =α _O *S of the corresponding similar expression of the model T.

In particular, after optimizing the smooth transition between patches and patches, the optimization equation for expression similarity estimation is obtained as follows:

E _E =E _R +E _O

Where E _R is the reprojection error optimization equation, L _i is all the feature points on the i-th patch, x _{v, i} is the v-th feature point on the i-th patch is defined by the formula of the local mixed deformation model above, p _{v, i} is the two-dimensional coordinates in the image corresponding to the feature point, P _i is the projection matrix of the i-th patch, and λ _R is the weight value of E _R ;

E _O is an equation that constrains the shape of adjacent patches, where v∈S is the vertex of the adjacent overlapping area, and E _O constrains the point of the overlapping part between the i-th patch and the adjacent j-th patch distance as close as possible.

Beneficial effect:

1. First of all, the present invention effectively shortens the time required for similarity estimation in the later stage through the selection of base expressions, and has the characteristics of high efficiency, high fidelity, and universalization;

2. Secondly, through the parameter conversion matrix, the dynamic realism of the current model face is guaranteed, with high fidelity;

3. Again, due to the similarity estimation and parameter conversion matrix adopted, the whole process has strong general applicability. Users of different ages, genders, and races can realize expression transfer in real time and with high fidelity;

4. Finally, the present invention optimizes the smooth transition process between patches, making expression transfer more realistic.

Description of drawings

Fig. 1 is the flow chart of the efficient expression transfer method that adopts basic expression space transformation of the present invention;

Fig. 2 is a schematic diagram of face model segmentation and feature points used for similarity estimation in the present invention.

detailed description

The present invention will be described in detail below with reference to the accompanying drawings and examples.

The invention provides an efficient expression transfer method using base expression space transformation

The process flow of the present invention is shown in Figure 1. The expression transfer method from 2D photos to 3D models proposed by the present invention is realized through two parts. First is the preprocessing part, as shown in the lower part of Figure 1, including: Step 1: Obtain the source model O, the source model O is a base expression model, which includes expression grids under multiple different expressions;

The basic expression model adopted by the source model in the present invention, such as Blendshape, obtains different facial expression models and builds them into a series of grid models with the same topological structure, and uses weight interpolation to interpolate and fuse different expressions according to weights , so that the face model can be transformed between multiple different expressions. In general, each Blendshape is used to express the transformation of an area of the face. During fusion, each Blendshape will not interfere with the deformation of other Blendshapes. To obtain the weight of each Blendshape, you only need to pay attention to the changes in the corresponding local area. That's it. The current mainstream way to build Blendshape is to refer to the FACS system and model 46 expressions as the base expressions of Blendshape.

Blendshape generally has two forms, one is Absolute Blendshape (Absolute BlendshapeModel), which adds all expressions according to a certain weight, and realizes deformation between different expressions by adjusting the weight, as described in formula (1), assuming that in addition to neutral expressions There are a total of K expressions, where α _k is the weight parameter of the k-th expression, b _k is the shape of the k-th base expression, and e is the weighted and fused expression shape. But this method can only deal with the fusion of different expressions. When multiple expressions need to exist at the same time, the correct effect cannot appear; the other is offset (difference, compensation) (Delta Blendshape Model), by selecting an The grid model of the expression is neutral, and the offset (difference) of other expressions relative to the neutral constitutes a Blendshape. The offset Blendshape can intuitively reflect the superposition effect of different expressions, which is described by formula (2), where u=b ₀ is the selected neutral expression, d _k =b _k -b ₀ is the offset of the kth expression relative to the neutral expression. This method is widely used. The offset Blendshape is used in this invention.

Step 2: Reconstruct facial expressions according to the source model O, and obtain expression description parameters of various facial expressions under the source model O;

The present invention adopts a single-view facial expression reconstruction method to perform facial expression reconstruction, which is also an important link in the present invention to realize facial expression transfer. Fig. 2 is a schematic diagram of face model segmentation and subsequent feature points used for similarity estimation. The feature points are symmetrically and evenly distributed, ensuring that each patch has several feature points, and the displacement direction of the feature points under different expressions is used as the description of the current expression. First, a set of face feature points needs to be defined. The feature points are symmetrically and evenly distributed, ensuring that each face has several feature points, and finding the corresponding vertex number in the 3D model. For any input facial expression photo, the two-dimensional coordinates of the defined marker points are detected, and the optimization equation is established so that the distance between the projected position of the feature point in the model and the position of the 2D point in the image is the smallest. The optimization formula (energy equation) is as follows

E＝∑||P(x _v )-p _v || ² (3)

Where P is the projection matrix, x _v is the three-dimensional position of the vth facial feature point, which can be obtained by formula (3), and p _v is the projection coordinate of the corresponding vth feature point on the two-dimensional plane.

Step 3, perform facial expression reconstruction on the target model T,

Obtain the expression description parameters of the two; perform similarity estimation according to the expression description parameters, and obtain the expression parameter conversion matrix and similarity matrix;

Facial expression transfer is to transfer the expression on the face of the source person to the face of the target person, so that the target person's face has the same expression as the source person's face. A common 3D expression transfer method is to directly transfer the Blendshape parameters describing the expression, but this requires the source and target models to use the same method to build Blendshape, the base expressions used have the same meaning (semantics), and one by one correspond. This requires creating a set of base expressions with consistent expressions when creating the model, and the final migration effect is also closely related to the similarity of the base expressions. This method not only requires a lot of labor from the artist, but also has many restrictions on the model. Still others achieve transfer by finding a linear mapping space between the target and source models, or by finding similar expressions in sequences. These methods all need to find the corresponding expressions between the source and the target, that is, to perform similarity estimation. At present, the methods for similarity estimation mainly include methods based on feature points in the image, based on model vertices, and based on triangular faces. Methods based on feature points or model vertices basically use the offset direction and size of comparison points for similarity estimation. Based on the triangular face, the similarity expression is generated by finding the corresponding triangular face of the model and transforming the direction of the corresponding triangular face. However, this method does not work well when the shape of the target face and the source face are relatively different.

In this step, assuming that for the i-th patch, under different expressions, the offset of the feature point relative to the neutral expression is δ _i , which is used as the description of the current expression, then the expression gap between the source model O and the target model T is given by the formula ( 4) Express.

Where δ _Oi is the expression description of the i-th patch of model O, and δ _Ti is the expression description of the i-th patch of model T. However, E _Ci only constrains the direction of feature point offset, and does not constrain the distance of feature point offset. Only this item cannot obtain accurate facial similarity estimation. The present invention adds a reference shape as the second constraint , formula (5).

For each expression in model O, the present invention obtains a corresponding expression of model T, and after aligning the expression with the neutral expression using the Platts analysis method, calculates the offset of the model vertices relative to the neutral expression, δ _{Di, v} is the offset of the v-th vertex in the i-th patch after the deformation transfer (deformation transfer), and Vi is all the vertices of the i-th patch.

The final optimization equation for the expression similarity estimation of each face is formula (6).

E _Si ＝λ _M *E _Di *E _Ci +λ _D *E _Di +λ _C *E _Ci (6)

By solving the above equations for all the faces, for each expression in the model O, a corresponding similar expression of a model T can be obtained, and all similar expressions can form a set of equivalent mixed deformation expression parameter conversion matrices. The expression parameters of the model T corresponding to the equivalent mixed deformation form a similarity matrix S for the mixed deformation parameters of each patch. Through the similarity matrix S, the expression parameter α _O of any model O can be transformed into The expression parameter α _T =α _O *S of the similar expression of the corresponding model T.

The optimization equations for single-view facial expression reconstruction also need to be adjusted for the base expressions. Not only should the minimum reprojection distance of the feature points in each patch be considered, but also the smooth transition between patches should be considered. The adjusted optimization equation contains two parts, as shown in formula (7).

E _E =E _R +E _O (7)

E _O ＝λ _O ∑ _v∈S ∑ _{(i, j)∈I, i>j} ||x _{v, i} -x _{v, j} || ₂ (9)

Where E _R is the reprojection error optimization equation (energy), L _i is all the feature points on the i-th patch, x _{v, i} is the v-th feature point on the i-th patch by the above local mixed deformation model Formula definition, p _{v, i} is the two-dimensional coordinates in the image corresponding to the feature point, P _i is the projection matrix of the i-th patch, and λ _R is the weight value of E _R.

E _O is the equation (energy) that constrains the shape of adjacent patches, where v∈S is the vertex of the adjacent overlapping area, and E _O constrains the overlap between the i-th patch and the adjacent j-th patch Some points are as close as possible.

The other part is the facial expression transfer process, as shown in the upper part of Figure 1, which includes: Step 4: Through the input user O's face photo, the local mixed deformation model of user O, and the equivalent T mixed deformation model, perform For single-view facial expression reconstruction, the obtained set of hybrid deformation parameters are then passed together to form a similarity matrix for facial expression transfer.

Specifically, in the process of expression transfer from a 2D image to a 3D model, input the picture of the character corresponding to model O, and detect the feature points. According to the method of formula (8), use the feature points of model O and the detected feature points to calculate E _R , divide the equivalent mixed deformation of the model T in the same way, bring it into E _O in the formula (9) for smooth constraints, and obtain the expression parameter α _O by solving E _E , and transform it into S through the similarity matrix is α _T , and the similar expressions of the transferred model T can be obtained by fusing the separated patches.

The invention proposes a facial expression transfer method using base expression space transformation. Based on the muscle distribution of the face, the facial area is segmented to select the base expression, and then the method of facial expression similarity estimation through the offset direction of the vertices is used to estimate the similarity of the base expressions between different models, and obtain A set of equivalent transformation matrices for blend shape and expression parameters is developed. The migration process of the present invention is carried out on the basis of facial expression reconstruction. By detecting facial feature points and calculating the reprojection error, facial expression reconstruction is realized, and the facial model segmented by equivalent blending deformation constraints has a complete Facial expressions, and finally obtain the hybrid deformation parameters of the target model through the parameter transformation matrix, and generate the transferred target expressions.

Compared with the facial expression transfer method, the present invention has three major advantages: high efficiency, high fidelity, and universalization. Firstly, the present invention effectively shortens the time required for later similarity estimation through the selection of base expressions. Secondly, through the parameter conversion matrix, the dynamic realism of the current model face is guaranteed, with high fidelity. Finally, due to the similarity estimation and parameter transformation matrix adopted, the whole process has strong general applicability. Users of different ages, genders, and races can realize expression transfer in real time and with high fidelity.

To sum up, this method effectively transfers the time cost and economic cost from 2D image to 3D facial model expression, improves the realism of virtual human expression and action, and has the potential of wide application.

To sum up, the above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

For those skilled in the art, it is obvious that the embodiments of the present invention are not limited to the details of the above-mentioned exemplary embodiments, and that the embodiments of the present invention can be implemented in other specific forms without departing from the spirit or essential features of the embodiments of the present invention. example. Therefore, no matter from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the embodiments of the present invention is defined by the appended claims rather than the above description, so it is intended that the All changes within the meaning and range of equivalents to the claims are included in the embodiments of the present invention. Any reference sign in a claim should not be construed as limiting the claim concerned. In addition, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Multiple units, modules or devices stated in system, device or terminal claims may also be realized by the same unit, module or device through software or hardware. The words first, second, etc. are used to denote names without implying any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention and not to limit them. Although the embodiments of the present invention have been described in detail with reference to the above preferred embodiments, those of ordinary skill in the art should understand that they can Modifications or equivalent replacements to the technical solutions of the embodiments of the present invention should not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (6)

1. An efficient expression migration method adopting a base expression space transformation is characterized by comprising the following steps: a preprocessing stage and a facial expression migration stage; the pre-treatment stage comprises:

acquiring a source model O, wherein the source model O is a base expression model and comprises expression grids under various different expressions;

performing facial expression reconstruction on the source model O and the target model T to obtain expression description parameters of the source model O and the target model T; performing similarity estimation according to the expression description parameters to obtain an expression parameter conversion matrix and a similarity matrix;

the facial expression migration stage comprises: and inputting the role picture into a base expression model, calculating expression description parameters of the target expression model according to the expression description parameters in the base expression model and the similarity matrix, and finishing expression migration.

2. The method of claim 1, wherein reconstructing the facial expression comprises: defining a group of human face characteristic points, and finding corresponding vertex serial numbers in the three-dimensional model; for any input facial expression photo, detecting two-dimensional coordinates of defined feature points, and establishing an optimization equation to enable the distance between the position of the feature points in the model projected onto the plane and the position of the 2D points in the image to be minimum; wherein the optimization equation is represented as follows:

E＝∑||P(x _v )-p _v || ₂

where P is the projection matrix, x _v For the three-dimensional position of the v-th facial feature point, p can be found from the optimization equation _v Is the projection coordinate of the corresponding v-th feature point on the two-dimensional plane.

3. The method for migrating expressions efficiently by using the spatial transformation of the base expression according to claim 1 or 2, wherein the source model O selects a grid model without expressions as a neutral reference, and the offsets of the rest of expressions relative to the neutral reference constitute the superposition effect of different expressions, and the offsets constitute the expression description parameters of different facial expressions.

4. The method of claim 3, wherein for the ith patch, under different expressions, the shift of feature points from neutral expressions is δ _i As a description of the current expression, the expression gap between the source model O and the target model T is represented by the following formula:

wherein E _Ci Constraining the direction of characteristic point offset between the source model O and the target model T; e _Di The distance of the characteristic point offset between the source model O and the target model T is constrained;

δ _Oi is an expressive description, δ, of the ith patch of the source model O _Ti Expression description of the ith patch of the model T; delta _Di，v Then the offset of the V-th vertex, V, in the ith patch after the modified pass is performed _i All vertices of the ith patch.

5. The method for efficient expression migration using expression-based spatial transformation according to claim 4, wherein the expression similarity estimation of each patch has the following optimization equation:

E _Si ＝λ _M *E _Di *E _Ci +λ _D *E _Di +λ _C *E _Ci

wherein E _Ci Is the difference in expression between the source model O and the target model T, E _Di Is to add a reference shape as a second term constraint to obtain an accurate face similarity estimate, λ _M 、λ _D 、λ _C Is a weighted value;

by solving the above equation for all patches, for each expression in the source model O, a corresponding similar expression of a target model T can be obtained, and all similar expressions can form a set of expression parameter transformation matrices of equivalent mixed deformation, for the expression parameters of the target model T corresponding to the expression parameter transformation matrices of equivalent mixed deformation, for the mixed deformation parameters of each patch, a similarity matrix S is formed, by which,the expression parameter alpha of any one source model O can be converted _O Expression parameter alpha of similar expression of corresponding model T is converted into _T ＝α _O *S。

6. The method for efficient expression migration using spatial transformations of basal expressions according to claim 5, wherein after optimizing the smooth transitions between patches, the optimization equation for expression similarity estimation is obtained as follows:

E _E ＝E _R +E _O

wherein E _R Optimizing the equation for reprojection error, L _i Is all the characteristic points, x, on the ith panel _v，i For the v characteristic point on the ith patch, defined by the formula of the local mixed deformation model above, p _v，i For two-dimensional coordinates, P, in the image corresponding to the feature point _i Is the projection matrix of the ith patch, λ _R Is E _R The weight value of (1);

E _O to constrain the shape of adjacent patches, where v ∈ S is the vertex of the adjacent overlap region, E _O The distance of the overlapping point between the ith patch and the adjacent jth patch is constrained to be as close as possible.

Images