CN115631318A - Method and device for acquiring same-topology grid data of multi-expression face and electronic equipment - Google Patents

Abstract

The application relates to a method and a device for acquiring same-topology grid data of a multi-expression face and electronic equipment, wherein the method comprises the following steps: obtaining dense semantic information of the face from an original face database, calculating first dense optical flows between different expression visual angle images at each visual angle, deforming vertexes of each expression grid point by point based on the first dense optical flows and a multi-visual angle positioning method to obtain expression grids with primary semanteme consistency enhancement, further calculating second dense optical flows between UV images of each expression grid, and correcting coordinates of vertexes of each expression grid based on the second dense optical flows to obtain same-topology grid data of the multi-expression face with strong semanteme consistency. Therefore, semantic information is obtained from the original face acquisition data, the semantic relation among expressions is constructed through the optical flow, the multi-expression grid data which are registered independently are deformed, the semantic consistency of the multi-expression grid data is enhanced, and the problems that the semantics among the expressions are inconsistent and the like due to the fact that the multi-expression face grid data which are obtained in the independent registration mode are solved.

Description

Method, device and electronic equipment for acquiring multi-expression face and topological grid data

technical field

The present application relates to the technical field of computer vision, and in particular to a method, device and electronic equipment for acquiring multi-expression face and topological grid data.

Background technique

High-quality 3D face and topological grid data play an important role in the production of games and animations in the industry, as well as in the provision of face datasets in the scientific research world.

In related technologies, most of the three-dimensional face and topological grid data are obtained by fitting a specified template face grid to the originally collected face scan, that is, through separate registration.

However, this registration method is independent of expressions, because in the process of registering these grid data, the original collected 3D face scans and corresponding multi-view images contain dense semantic information of different expressions, therefore, separate registration Although the method can achieve the same topology, the same area of the grid may correspond to different semantics of the face, that is, the semantic consistency between expressions cannot be achieved.

Therefore, it is necessary to establish a registration method for multi-expression face grid data.

Contents of the invention

This application provides a method, device, and electronic equipment for acquiring multi-expression face grid data with the same topology to solve the problem of semantic inconsistency between expressions in the multi-expression face grid data of the same individual obtained through separate registration.

The embodiment of the first aspect of the present application provides an acquisition of multi-expression face and topological grid data, including the following steps:

Obtain face-intensive semantic information from the original face database;

Based on the semantic information, calculate the first dense optical flow between different expression view graphs in each viewing angle, and deform the vertices of each expression grid vertex by vertex based on the first dense optical flow and multi-view positioning method to obtain preliminary semantics Consistently enhanced expression meshes; and

Calculate the second dense optical flow between the UV maps of each expression grid based on the preliminary semantically consistent enhanced expression grid, and correct the coordinates of the vertices of each expression grid based on the second dense optical flow to obtain Multi-expression face with strong semantic consistency and topological grid data.

According to an embodiment of the present application, the first dense optical flow between different expression view maps under each view is calculated based on the semantic information, and the first dense optical flow and the multi-view positioning method are used to deform each The vertices of the expression grid get a preliminary semantically consistent enhanced expression grid, including:

Projecting the first expression grid to the expression graph under multiple viewing angles to obtain the two-dimensional position of each vertex of the first expression grid in the first expression graph;

Calculate the first dense optical flow from the first expression map to the second expression image through the first optical flow solver, and obtain the semantically consistent constraints of each vertex of the first expression grid according to the first dense optical flow down-projecting to the two-dimensional position of the second emoticon;

Using the topological consistency of the first expression grid and the second expression grid, obtain the two-dimensional position of each vertex of the second expression grid projected to the second expression map under the constraint of semantic consistency;

Using the multi-view positioning method, reposition each vertex of the second expression grid to obtain a deformed second expression grid, and combine the first expression grid and the deformed second expression grid as a preliminary semantically consistent augmented expression grid.

According to an embodiment of the present application, the expression grid based on the preliminary semantic consistency enhancement calculates the second dense optical flow between the expression grid UV (UV texture map coordinates) maps, and based on the second dense Optical flow corrects the coordinates of the vertices of each expression grid, including:

respectively projecting the first expression grid and the deformed second expression grid onto the corresponding original 3D face scan, to obtain the three-dimensional coordinates and material information;

Utilizing the topological consistency, projecting the first expression grid and the deformed second expression grid into the same UV space, and generating a UV map and a three-dimensional coordinate distribution based on the three-dimensional coordinates and the material information field;

A second dense optical flow from the UV map of the first expression grid to the UV map of the deformed second expression grid is calculated by a second optical flow solver.

According to an embodiment of the present application, the correcting the coordinates of the vertices of the expression grids based on the second dense optical flow includes:

Acquiring the UV coordinates of each vertex of the first expression grid on the UV map of the deformed second expression grid under the constraint of semantic consistency according to the second dense optical flow;

Sampling from the three-dimensional coordinate distribution field of the deformed second expression grid to obtain the original three-dimensional face scan of the deformed second expression grid under the constraints of semantic consistency of each vertex of the first expression grid The three-dimensional coordinates of the figure;

Correcting the deformed second expression grid based on the coordinates of each vertex of the first expression grid in the original three-dimensional face scan image of the deformed second expression grid.

According to an embodiment of the present application, both the first optical flow solver and the second optical flow solver are trained using a pre-trained neural network for optical flow calculation.

According to the method for acquiring multi-expression face and topological grid data according to the embodiment of the present application, the dense semantic information of the face is obtained from the original face database, and the first dense optical flow between different expression view maps under each view is calculated , and based on the first dense optical flow and multi-view positioning method, the vertices of each expression grid are deformed vertex by vertex, and the expression grid with preliminary semantic consistency enhancement is obtained, and then the second dense optical flow between the UV maps of each expression grid is calculated. And based on the second dense optical flow, the coordinates of the vertices of each expression grid are corrected, and the multi-expression face and topological grid data with strong semantic consistency are obtained. Therefore, the semantic information is obtained from the original face collection data, the semantic relationship between expressions is constructed by optical flow, and the separately registered multi-expression grid data is deformed to enhance its semantic consistency, which solves the problem of simultaneous expression obtained through separate registration. The grid data of individual multi-expression face makes the semantic inconsistency between expressions and other issues.

The embodiment of the second aspect of the present application provides a device for acquiring multi-expression face and topological grid data, including:

The acquisition module is used to acquire face-intensive semantic information from the original face database;

The first calculation module is used to calculate the first dense optical flow between different expression view graphs under each view angle based on the semantic information, and deform each expression network vertex by vertex based on the first dense optical flow and the multi-view positioning method vertices of the grid, resulting in a preliminary semantically consistent enhanced expression grid; and

The second calculation module is used to calculate the second dense optical flow between the UV maps of the expression grids based on the preliminary semantically consistent enhanced expression grid, and correct the expression grids based on the second dense optical flow. The coordinates of the vertices of the grid are used to obtain the multi-expression face and topological grid data with strong semantic consistency.

According to an embodiment of the present application, the first calculation module is specifically used for:

Projecting the first expression grid to the expression graph under multiple viewing angles to obtain the two-dimensional position of each vertex of the first expression grid in the first expression graph;

Calculate the first dense optical flow from the first expression map to the second expression image through the first optical flow solver, and obtain the semantically consistent constraints of each vertex of the first expression grid according to the first dense optical flow down-projecting to the two-dimensional position of the second emoticon;

Using the topological consistency of the first expression grid and the second expression grid, obtain the two-dimensional position of each vertex of the second expression grid projected to the second expression map under the constraint of semantic consistency;

Using the multi-view positioning method, reposition each vertex of the second expression grid to obtain a deformed second expression grid, and combine the first expression grid and the deformed second expression grid as a preliminary semantically consistent augmented expression grid.

According to an embodiment of the present application, the second calculation module is specifically used for:

respectively projecting the first expression grid and the deformed second expression grid onto the corresponding original 3D face scan, to obtain the three-dimensional coordinates and material information;

Utilizing the topological consistency, projecting the first expression grid and the deformed second expression grid into the same UV space, and generating a UV map and a three-dimensional coordinate distribution based on the three-dimensional coordinates and the material information field;

A second dense optical flow from the UV map of the first expression grid to the UV map of the deformed second expression grid is calculated by a second optical flow solver.

According to an embodiment of the present application, the second calculation module is specifically used for:

Acquiring the UV coordinates of each vertex of the first expression grid on the UV map of the deformed second expression grid under the constraint of semantic consistency according to the second dense optical flow;

Sampling from the three-dimensional coordinate distribution field of the deformed second expression grid to obtain the original three-dimensional face scan of the deformed second expression grid under the constraints of semantic consistency of each vertex of the first expression grid The three-dimensional coordinates of the figure;

Correcting the deformed second expression grid based on the coordinates of each vertex of the first expression grid in the original three-dimensional face scan image of the deformed second expression grid.

According to an embodiment of the present application, both the first optical flow solver and the second optical flow solver are trained using a pre-trained neural network for optical flow calculation.

According to the acquisition device of multi-expression face and topological grid data according to the embodiment of the present application, the dense semantic information of the face is obtained from the original face database, and the first dense optical flow between different expression view maps under each view is calculated , and based on the first dense optical flow and multi-view positioning method, the vertices of each expression grid are deformed vertex by vertex, and the expression grid with preliminary semantic consistency enhancement is obtained, and then the second dense optical flow between the UV maps of each expression grid is calculated. And based on the second dense optical flow, the coordinates of the vertices of each expression grid are corrected, and the multi-expression face and topological grid data with strong semantic consistency are obtained. Therefore, the semantic information is obtained from the original face collection data, the semantic relationship between expressions is constructed by optical flow, and the separately registered multi-expression grid data is deformed to enhance its semantic consistency, which solves the problem of simultaneous expression obtained through separate registration. The grid data of individual multi-expression face makes the semantic inconsistency between expressions and other issues.

The embodiment of the third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor executes the program to realize The method for acquiring multi-expression human face and topological grid data as described in the above-mentioned embodiments.

The embodiment of the fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to realize multi-expression human face and topological grid data as described in the above-mentioned embodiment method of obtaining .

The method for obtaining multi-expression human face and topological grid data in the embodiment of the present application has the following advantages:

(1) Ability to obtain registered face grid data with strong semantic consistency.

(2) This registration method can be used as an enhanced method of the traditional registration method. It can not only directly operate on the face grid data registered with the traditional method, but also enhance the semantic consistency between expressions.

(3) This registration method can directly use the original collected face data without adding additional data, and is easy to operate.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a flow chart of a method for acquiring multi-expression human face and topological grid data according to an embodiment of the present application;

2 is a schematic block diagram of a device for acquiring multi-expression human face and topological grid data according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary, and are intended to explain the present application, and should not be construed as limiting the present application.

The method, device and electronic equipment for acquiring multi-expression human face and topological grid data according to the embodiments of the present application will be described below with reference to the accompanying drawings. Aiming at the problem of semantic inconsistency among facial expressions of the multi-expression facial grid data of the same individual obtained through separate registration mentioned in the above-mentioned background technology, this application provides an acquisition of multi-expression human face with topological grid data method, in this method, the dense semantic information of the face is obtained from the original face database, and the first dense optical flow between different expression view maps under each viewing angle is calculated, and based on the first dense optical flow and multi-view positioning The method deforms the vertices of each expression grid vertex by vertex, and obtains the expression grid with enhanced semantic consistency, and then calculates the second dense optical flow between the UV maps of each expression grid, and corrects each expression grid based on the second dense optical flow The coordinates of the vertices of the multi-expression face and the topological grid data with strong semantic consistency are obtained. As a result, the problem of semantic inconsistency among expressions in the multi-expression face grid data of the same individual obtained through separate registration is solved, and the corresponding relationship between semantic information between expressions is constructed through optical flow. The data is transformed to enhance its semantic consistency.

Specifically, FIG. 1 is a schematic flow chart of a method for acquiring multi-expression face and topological grid data provided by an embodiment of the present application.

As shown in Figure 1, the method for obtaining multi-expression human face with topological grid data includes the following steps:

In step S101, face-intensive semantic information is obtained from the original face database.

It should be understood that high-quality 3D face and topological grid data play an important role in the production of games and animations in the industry, and in the provision of face datasets in the scientific research community. Generally speaking, the multi-expression face grid data of the same individual is obtained through separate registration, but this method ignores the semantic consistency relationship between expressions.

Therefore, in order to obtain multi-expression face and topological grid data with strong semantic consistency, the embodiment of the present application can first scan through a three-dimensional method, and collect face-intensive semantic information from original face databases such as multi-view images.

In step S102, based on the semantic information, calculate the first dense optical flow between different expression view graphs in each view, and deform the vertices of each expression grid vertex by vertex based on the first dense optical flow and the multi-view positioning method to obtain a preliminary Semantically consistent augmented expression grids.

Further, in some embodiments, the first dense optical flow between different expression view graphs under each view is calculated based on the semantic information, and the vertices of each expression grid are deformed vertex by vertex based on the first dense optical flow and the multi-view positioning method , to obtain a preliminary semantically consistent enhanced expression grid, including: projecting the first expression grid to the expression map under multiple perspectives, and obtaining the two-dimensional positions of each vertex of the first expression grid in the first expression map; through the first The optical flow solver calculates the first dense optical flow from the first expression image to the second expression image, and according to the first dense optical flow, obtains the second expression of each vertex of the first expression grid projected to the second expression image under the constraint of semantic consistency. dimensional position; using the topological consistency of the first expression grid and the second expression grid, each vertex of the second expression grid is projected to the two-dimensional position of the second expression map under the constraints of semantic consistency; using the multi-view positioning method , relocate each vertex of the second expression grid to obtain the deformed second expression grid, and use the first expression grid and the deformed second expression grid as the preliminary semantically consistent enhanced expression grid.

Wherein, the first expression grid can be the only original expression grid, and the second expression grid can be a corresponding multi-view expression grid, for example, the first expression grid is A expression grid, and the second expression grid It can only represent B expression grid, or can simultaneously represent B expression grid and C expression grid, or can simultaneously represent B expression grid, C expression grid and D expression grid, or even more. Specifically, through the face-intensive semantic information collected from the original face database such as multi-view pictures, the semantic correspondence between expressions can be established using the streamer method. If the strong semantics of the grids between expressions are consistent, then these grids will be projected to different perspective pictures under each perspective, so that the same grid area corresponds to the same face area, otherwise there will be a certain offset. Based on the formed offset, the correct position of each vertex of the grid under the constraints of semantic consistency can be obtained, and then the mesh is deformed vertex by vertex by multi-view positioning to obtain a mesh with preliminary semantic consistency enhancement.

For example, suppose an individual has two expressions, A and B, among which, expression A is the only original expression, and expression B is the corresponding multi-view expression. For example, the semantics of B expression grid can be mapped to A expression grid.

Specifically, first of all, in the embodiment of the present application, the expression grid of A can be projected onto the expression maps under various viewing angles according to the camera parameters (the camera parameters can be provided with the original collected data, or can be obtained through registration), so as to obtain The two-dimensional position of the vertices of the A expression grid in the A expression image; secondly, use the optical flow solver based on the neural network to calculate the dense optical flow from the A expression image to the B expression image under each viewing angle, that is, the first dense optical flow; Again, for each vertex of the expression grid of A, the two-dimensional position of the vertex on the expression map of B is obtained according to the optical flow calculation results, and the topological consistency of the expression grids of A and B can be used to directly obtain the position of each vertex of the expression B. Under the constraints of consistent semantics, it is projected to the correct position under each view of the B expression grid; finally, using multi-view positioning, the position of each vertex of the B expression grid is repositioned, and the deformed B expression grid is obtained. grid, its semantics should be coarsely aligned with the A expression grid.

In step S103, based on the preliminary semantically consistent enhanced expression grid, calculate the second dense optical flow between the UV maps of each expression grid, and correct the coordinates of the vertices of each expression grid based on the second dense optical flow, to obtain strong Semantically consistent multi-expression face with topological grid data.

Further, in some embodiments, the second dense optical flow between the UV maps of each expression grid is calculated based on the preliminary semantically consistent enhanced expression grid, and the coordinates of the vertices of each expression grid are corrected based on the second dense optical flow , including: respectively projecting the first expression grid and the deformed second expression grid onto the corresponding original 3D face scan, and obtaining the 3D coordinates and material information of each vertex projected onto the original 3D face scan; Using topological consistency, project the first expression grid and the deformed second expression grid to the same UV space, and generate a UV map and a 3D coordinate distribution field based on the 3D coordinates and material information; calculate through the second optical flow solver The second dense optical flow from the UV map of the first expression mesh to the UV map of the deformed second expression mesh.

Further, in some embodiments, correcting the coordinates of the vertices of each expression grid based on the second dense optical flow includes: obtaining the deformed coordinates of each vertex of the first expression grid under the constraint of semantic consistency according to the second dense optical flow The UV coordinates on the UV map of the second expression grid; the second expression network after each vertex of the first expression grid is deformed under the semantically consistent constraint is obtained by sampling from the three-dimensional coordinate distribution field of the deformed second expression grid The three-dimensional coordinates of the original three-dimensional face scan of the grid; the second expression grid after deformation is corrected based on the coordinates of the original three-dimensional face scan of the second expression grid after the deformation of each vertex of the first expression grid.

Specifically, in daily life, cameras are mostly used to collect face-intensive semantic information. However, due to some camera parameter errors and other problems, in order to obtain a stronger semantically consistent expression grid, 3D face scanning can be used In this way, the expression grid is projected onto the respective original 3D face scans, the 3D projection position and material of the grid on the scan are obtained, and the original expression grid and the deformed expression grid are projected to the Generate a UV map on the same UV space, that is, project the first expression grid and the deformed second expression grid to the same UV space, and generate a UV map based on the three-dimensional coordinates and material information. In the case of consistent semantics, each expression The parts of the face area in the UV map of the grid should coincide, otherwise there will be a certain offset.

Furthermore, under the condition of fine semantic alignment, streamer can also be used to correct the offset, and then obtain the correct three-dimensional projection position of the expression grid in the original face scan under the constraint of semantic consistency, thereby correcting each expression The vertex positions of the mesh, and finally a strongly semantically consistent mesh.

It should be noted that both the first optical flow solver and the second optical flow solver used in the embodiment of the present application are trained by using a pre-trained neural network for optical flow calculation. For example, based on the facial expressions of A and B obtained in the above steps, first, the grids of the facial expressions of A and B are projected onto their respective original 3D face scans, so as to obtain the 3D coordinates and material information of each vertex projected onto the scan ;Secondly, using topological consistency, project A and B expression grids into the same UV space, and use the 3D coordinates and material information obtained above to generate UV maps and 3D coordinate distribution fields; again, use neural network-based optical flow The solver calculates the dense optical flow from the UV map of expression A to the UV map of B expression; finally, for each vertex of A expression grid, the UV coordinates of the vertex on the B expression UV map are obtained according to the optical flow calculation results, and directly from B The coordinates of the vertices in the original 3D face scan of the expression B are sampled from the three-dimensional coordinate distribution field of the expression, and the correct position of each vertex of the expression B grid on the face scan is obtained by using topological consistency.

Therefore, the correct position of each vertex obtained through the above steps further modifies the B expression grid, so as to obtain the B expression grid data finely aligned with the A expression grid.

In summary, the method for acquiring multi-expression face and topological grid data in the embodiment of the present application can obtain semantic information from original collected face data such as 3D face scans and multi-view pictures, and use the optical flow method to establish the relationship between expressions. Semantic correspondence, according to this relationship, determine the correct position of each vertex of the expression face grid, and further deform the grid to enhance the semantic consistency between expressions; that is, the embodiment of the present application can use the rough semantic alignment step and the fine semantic The alignment step realizes the acquisition of strongly semantically consistent multi-expression faces with topological grid data.

According to the method for acquiring multi-expression face and topological grid data according to the embodiment of the present application, the dense semantic information of the face is obtained from the original face database, and the first dense optical flow between different expression view maps under each view is calculated , and based on the first dense optical flow and multi-view positioning method, the vertices of each expression grid are deformed vertex by vertex, and the expression grid with preliminary semantic consistency enhancement is obtained, and then the second dense optical flow between the UV maps of each expression grid is calculated. And based on the second dense optical flow, the coordinates of the vertices of each expression grid are corrected, and the multi-expression face and topological grid data with strong semantic consistency are obtained. Therefore, the semantic information is obtained from the original face collection data, the semantic relationship between expressions is constructed by optical flow, and the separately registered multi-expression grid data is deformed to enhance its semantic consistency, which solves the problem of simultaneous expression obtained through separate registration. The grid data of individual multi-expression face makes the semantic inconsistency between expressions and other issues.

Next, the device for acquiring multi-expression human face and topological grid data according to the embodiment of the present application will be described with reference to the accompanying drawings.

FIG. 2 is a schematic block diagram of a device for acquiring multi-expression face and topological grid data according to an embodiment of the present application.

As shown in FIG. 2 , the device 10 for acquiring multi-expression face and topological grid data includes: an acquisition module 100 , a first calculation module 200 and a second calculation module 300 .

Wherein, the obtaining module 100 is used to obtain face-intensive semantic information from the original face database;

The first calculation module 200 is used to calculate the first dense optical flow between different expression view graphs under each viewing angle based on the semantic information, and deform the vertices of each expression grid vertex by vertex based on the first dense optical flow and the multi-view positioning method, obtain a preliminary semantically consistent augmented expression grid; and

The second calculation module 300 is used to calculate the second dense optical flow between the UV maps of each expression grid based on the preliminary semantically consistent enhanced expression grid, and correct the coordinates of the vertices of each expression grid based on the second dense optical flow, The multi-expression face and topological grid data with strong semantic consistency are obtained.

Further, in some embodiments, the first calculation module 200 is specifically used for:

Projecting the first expression grid to the expression graph under multiple viewing angles to obtain the two-dimensional position of each vertex of the first expression grid in the first expression graph;

Calculate the first dense optical flow from the first expression map to the second expression image through the first optical flow solver, and obtain each vertex of the first expression grid according to the first dense optical flow and project it to the second expression under the constraint of semantic consistency The two-dimensional position of the graph;

Utilizing the topological consistency of the first expression grid and the second expression grid, each vertex of the second expression grid is projected to the two-dimensional position of the second expression map under the constraint of semantic consistency;

Using the multi-view positioning method, each vertex of the second expression grid is repositioned to obtain the deformed second expression grid, and the first expression grid and the deformed second expression grid are used as the initial semantic consistency enhancement Emoticon grid.

Further, in some embodiments, the second calculation module 300 is specifically used for:

Projecting the first expression grid and the deformed second expression grid onto the corresponding original 3D face scan, obtaining the 3D coordinates and material information of each vertex projected onto the original 3D face scan;

Using topological consistency, project the first expression grid and the deformed second expression grid to the same UV space, and generate a UV map and a three-dimensional coordinate distribution field based on the three-dimensional coordinates and material information;

The second dense optical flow from the UV map of the first expression grid to the UV map of the deformed second expression grid is calculated by the second optical flow solver.

Further, in some embodiments, the second calculation module 300 is specifically used for:

Obtaining the UV coordinates on the UV map of the second expression grid after each vertex of the first expression grid is deformed under the semantically consistent constraint according to the second dense optical flow;

Sampling from the three-dimensional coordinate distribution field of the deformed second expression grid to obtain the three-dimensional coordinates of the original three-dimensional human face scan of the second expression grid after each vertex of the first expression grid is deformed under the constraint of semantic consistency;

The deformed second expression grid is corrected based on the coordinates of each vertex of the first expression grid in the original three-dimensional face scan image of the deformed second expression grid.

Further, in some embodiments, both the first optical flow solver and the second optical flow solver are trained using a pre-trained neural network for optical flow calculation.

According to the acquisition device of multi-expression face and topological grid data according to the embodiment of the present application, the dense semantic information of the face is obtained from the original face database, and the first dense optical flow between different expression view maps under each view is calculated , and based on the first dense optical flow and multi-view positioning method, the vertices of each expression grid are deformed vertex by vertex, and the expression grid with preliminary semantic consistency enhancement is obtained, and then the second dense optical flow between the UV maps of each expression grid is calculated. And based on the second dense optical flow, the coordinates of the vertices of each expression grid are corrected, and the multi-expression face and topological grid data with strong semantic consistency are obtained. Therefore, the semantic information is obtained from the original face collection data, the semantic relationship between expressions is constructed by optical flow, and the separately registered multi-expression grid data is deformed to enhance its semantic consistency, which solves the problem of simultaneous expression obtained through separate registration. The grid data of individual multi-expression face makes the semantic inconsistency between expressions and other issues.

FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. This electronic equipment can include:

A memory 301 , a processor 302 , and a computer program stored in the memory 301 and executable on the processor 302 .

When the processor 302 executes the program, the method for acquiring multi-expression human face and topological grid data provided in the above-mentioned embodiments is implemented.

Further, the electronic equipment also includes:

The communication interface 303 is used for communication between the memory 301 and the processor 302 .

The memory 301 is used to store computer programs that can run on the processor 302 .

The memory 301 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

If the memory 301, the processor 302, and the communication interface 303 are implemented independently, the communication interface 303, the memory 301, and the processor 302 may be connected to each other through a bus to complete mutual communication. The bus may be an Industry Standard Architecture (Industry Standard Architecture, ISA for short) bus, a Peripheral Component Interconnect (PCI for short) bus, or an Extended Industry Standard Architecture (EISA for short) bus. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in FIG. 3 , but it does not mean that there is only one bus or one type of bus.

Optionally, in terms of specific implementation, if the memory 301, processor 302, and communication interface 303 are integrated on one chip, then the memory 301, processor 302, and communication interface 303 can communicate with each other through the internal interface.

The processor 302 may be a central processing unit (Central Processing Unit, referred to as CPU), or a specific integrated circuit (Application Specific Integrated Circuit, referred to as ASIC), or configured to implement one or more of the embodiments of the present application integrated circuit.

The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above method for acquiring multi-expression face and topological grid data is realized.

In the description of this specification, descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or N embodiments or examples in an appropriate manner. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a custom logical function or step of a process , and the scope of preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved, which shall It should be understood by those skilled in the art to which the embodiments of the present application belong.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment used. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connection with one or N wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as it may be possible, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable means if necessary. processing to obtain programs electronically and store them in computer memory.

It should be understood that each part of the present application may be realized by hardware, software, firmware or a combination thereof. In the above embodiments, the N steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: a discrete Logic circuits, ASICs with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed , including one or a combination of the steps of the method embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present application, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (10)

1. A method for acquiring multi-expression face same-topology grid data is characterized by comprising the following steps:

obtaining dense semantic information of the face from an original face database;

calculating first dense optical flows among different expression visual angle graphs under all visual angles based on the semantic information, and deforming the vertexes of all expression grids vertex by vertex based on the first dense optical flows and a multi-visual angle positioning method to obtain primary expression grids with consistent and enhanced semantics; and

and calculating second dense optical flows among the UV images of the expression grids based on the expression grids subjected to the preliminary semantic consistency enhancement, and correcting the coordinates of the vertexes of the expression grids based on the second dense optical flows to obtain multi-expression face homotopological grid data with consistent strong semantics.

2. The method of claim 1, wherein the calculating a first dense optical flow between different expression perspective images at each perspective based on the semantic information, and deforming the vertices of each expression mesh vertex by vertex based on the first dense optical flow and a multi-perspective positioning method to obtain a preliminary semantically consistent enhanced expression mesh comprises:

projecting a first expression grid to an expression graph under multiple visual angles to obtain two-dimensional positions of each vertex of the first expression grid on the first expression graph;

calculating a first dense optical flow from the first expression graph to a second expression graph through a first optical flow solver, and acquiring two-dimensional positions of each vertex of the first expression grid projected to the second expression graph under semantic consistency constraint according to the first dense optical flow;

obtaining two-dimensional positions of the second expression graph projected by each vertex of the second expression grid under semantic consistency constraint by using the topological consistency of the first expression grid and the second expression grid;

and repositioning each vertex of the second expression grid by using the multi-view positioning method to obtain a deformed second expression grid, and taking the first expression grid and the deformed second expression grid as the expression grid with the primary semantic consistency enhanced.

3. The method of claim 2, wherein the calculating a second dense optical flow between the expression mesh UV maps based on the preliminary semantically consistent enhanced expression mesh and modifying the coordinates of the vertices of each expression mesh based on the second dense optical flow comprises:

respectively projecting the first expression grid and the deformed second expression grid to corresponding original three-dimensional face scanning images to obtain three-dimensional coordinates and material information of each vertex projected to the original three-dimensional face scanning images;

projecting the first expression grid and the deformed second expression grid to the same UV space by using the topological consistency, and generating a UV graph and a three-dimensional coordinate distribution field based on the three-dimensional coordinate and the material information;

and calculating a second dense optical flow from the UV image of the first expression grid to the UV image of the deformed second expression grid through a second optical flow solver.

4. The method of claim 3, wherein said modifying the coordinates of the vertices of each expression mesh based on the second dense optical flow comprises:

acquiring UV coordinates of each vertex of the first expression grid on the UV image of the deformed second expression grid under the semantic consistency constraint according to the second dense optical flow;

sampling from the three-dimensional coordinate distribution field of the deformed second expression grid to obtain the three-dimensional coordinates of the original three-dimensional face scanning image of the deformed second expression grid under the semantic consistency constraint of each vertex of the first expression grid;

and correcting the deformed second expression grid based on the coordinates of each vertex of the first expression grid in the original three-dimensional face scan image of the deformed second expression grid.

5. The method of claim 4, wherein the first optical flow solver and the second optical flow solver are each trained using a pre-trained neural network for optical flow computation.

6. An apparatus for acquiring data of a multi-expression face same-topology grid, comprising:

the acquisition module is used for acquiring face dense semantic information from an original face database;

the first calculation module is used for calculating first dense optical flows among different expression visual angle graphs under all visual angles based on the semantic information, and deforming vertexes of all expression grids vertex by vertex based on the first dense optical flows and a multi-visual angle positioning method to obtain expression grids with primary semantic consistency enhancement; and

and the second calculation module is used for calculating second dense optical flows among the UV images of the expression grids based on the expression grids with the enhanced primary semantic consistency, and correcting the coordinates of the vertexes of the expression grids based on the second dense optical flows to obtain the data of the multi-expression face homotopology grids with the consistent strong semantic consistency.

7. The apparatus of claim 6, wherein the first computing module is specifically configured to:

projecting a first expression grid to an expression graph under multiple visual angles to obtain two-dimensional positions of each vertex of the first expression grid on the first expression graph;

calculating a first dense optical flow from the first expression graph to a second expression graph through a first optical flow solver, and acquiring two-dimensional positions of each vertex of the first expression grid projected to the second expression graph under semantic consistency constraint according to the first dense optical flow;

obtaining two-dimensional positions of the second expression graph projected by each vertex of the second expression grid under semantic consistency constraint by using the topological consistency of the first expression grid and the second expression grid;

and repositioning each vertex of the second expression grid by using the multi-view positioning method to obtain a deformed second expression grid, and taking the first expression grid and the deformed second expression grid as the expression grid with the primary semantic consistency enhanced.

8. The apparatus of claim 7, wherein the second computing module is specifically configured to:

respectively projecting the first expression grid and the deformed second expression grid onto corresponding original three-dimensional face scanning images to obtain three-dimensional coordinates and material information of the original three-dimensional face scanning images projected by each vertex;

projecting the first expression grid and the deformed second expression grid to the same UV space by using the topological consistency, and generating a UV graph and a three-dimensional coordinate distribution field based on the three-dimensional coordinate and the material information;

and calculating a second dense optical flow from the UV image of the first expression grid to the UV image of the deformed second expression grid through a second optical flow solver.

9. An electronic device, comprising: a memory, a processor and a computer program stored on the memory and operable on the processor, the processor executing the program to implement the method for acquiring multi-expression face homotopology grid data according to any one of claims 1 to 5.

10. A computer-readable storage medium on which a computer program is stored, the program being executed by a processor for implementing the method for acquiring multi-expression facial co-topological mesh data according to any one of claims 1 to 5.

Images