WO2024088061A1 - Face reconstruction and occlusion region recognition method, apparatus and device, and storage medium - Google Patents

Abstract

Embodiments of the present application provide a face reconstruction and occlusion region recognition method, apparatus and device, and a storage medium. The method comprises: inputting a face image into a first network structure, and outputting a reconstruction parameter vector of the face image during three-dimensional reconstruction by means of the first network structure; performing three-dimensional reconstruction of the face image on the basis of the reconstruction parameter vector to generate three-dimensional face information; performing rendering and mapping processing on the three-dimensional face information to generate two-dimensional face information comprising network segmentation data; acquiring a depth feature map extracted by the first network structure, and constructing map structure information on the basis of the depth feature map and the two-dimensional face information; and inputting the map structure information into a second network structure, and outputting an occlusion region corresponding to the face image by means of the second network structure. The present solution optimizes the overall resource deployment, can meet application scenarios having multiple parallel tasks and high real-time requirements, and achieves higher accuracy in occlusion region determination.

Description

Face reconstruction and occluded area recognition method, device, equipment and storage medium

This application claims priority to the Chinese patent application filed with the China Patent Office on October 27, 2022, with application number 202211328264.0, the entire contents of which are incorporated by reference into this application.

Technical Field

The embodiments of the present application relate to the field of image processing technology, and in particular to a method, device, equipment and storage medium for face reconstruction and occlusion area recognition.

Background technique

With the development and popularization of virtual reality technology, people are no longer satisfied with ordinary two-dimensional plane interactions, and have an urgent need for three-dimensional space applications such as 3D beautification and 3D face pinching. Currently, most 3D face reconstruction applications such as 3D beautification and 3D stylization are deployed in live broadcast, social networking and other scenarios, and have high requirements for real-time performance and reconstruction effects. At the same time, because the actual production environment is much more complicated than imagined, the input data does not always contain a complete face or have self-occlusion, object occlusion, etc., so the model needs to be robust to a variety of input data. At the same time, it is often necessary to determine the actual occluded area, so as to perform a series of post-processing operations.

In the related technology, convolutional neural networks are usually used to extract 3D face features and segment 2D faces for occlusion area determination. It is necessary to start two separate tasks for prediction, which requires more resource deployment and cannot meet the needs of some application scenarios with many parallel tasks and high real-time requirements. The occlusion area determination performed in this way has poor robustness. In addition, most existing occlusion area determination methods use traditional semantic segmentation tasks for processing, and the accuracy of occlusion area determination is low.

Summary of the invention

The embodiments of the present application provide a method, apparatus, device and storage medium for face reconstruction and occlusion area recognition, which solves the problems in the related technologies, optimizes the overall resource deployment, can meet the application scenarios with many parallel tasks and high real-time requirements, and has higher accuracy in occlusion area determination.

In a first aspect, an embodiment of the present application provides a method for face reconstruction and occlusion area recognition, the method comprising:

Inputting a face image into a first network structure, and outputting a reconstruction parameter vector of the face image when three-dimensional reconstruction is performed through the first network structure;

Performing three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information;

Rendering and mapping the three-dimensional face information to generate two-dimensional face information containing network segmentation data;

Obtaining a depth feature map extracted by the first network structure, and constructing graph structure information based on the depth feature map and the two-dimensional face information;

The graph structure information is input into a second network structure, and the occluded area corresponding to the face image is output through the second network structure.

In a second aspect, the embodiment of the present application further provides a face reconstruction and occlusion area recognition device, comprising:

A parameter vector generation module, configured to input a face image into a first network structure, and output a reconstruction parameter vector of the face image when three-dimensional reconstruction is performed through the first network structure;

a three-dimensional information determination module, configured to perform three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information;

A rendering and mapping module, configured to render and map the three-dimensional face information to generate two-dimensional face information containing network segmentation data;

A graph information construction module, configured to obtain a depth feature map extracted by the first network structure, and construct graph structure information based on the depth feature map and the two-dimensional face information;

The occlusion area determination module is configured to input the graph structure information into a second network structure, and output the occlusion area corresponding to the face image through the second network structure.

In a third aspect, the embodiment of the present application further provides a face reconstruction and occlusion area recognition device, the device comprising:

one or more processors;

a storage device for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the face reconstruction and occlusion area recognition method described in the embodiment of the present application.

In a fourth aspect, the embodiments of the present application further provide a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the embodiments of the present application when executed by a computer processor. The face reconstruction and occluded area recognition method described in the embodiment.

In the fifth aspect, an embodiment of the present application also provides a computer program product, which includes a computer program, which is stored in a computer-readable storage medium, and at least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the face reconstruction and occluded area recognition method described in the embodiment of the present application.

In an embodiment of the present application, a face image is input into a first network structure, a reconstruction parameter vector of the face image during three-dimensional reconstruction is output through the first network structure, and then three-dimensional reconstruction of the face image is performed based on the reconstruction parameter vector to generate three-dimensional face information, and the three-dimensional face information is rendered and mapped to generate two-dimensional face information containing network segmentation data. At this time, a depth feature map extracted by the first network structure is further obtained, and graph structure information is constructed based on the depth feature map and the two-dimensional face information. The graph structure information is input into a second network structure, and the occlusion area corresponding to the face image is output through the second network structure. The face reconstruction and occlusion area recognition method provided by this scheme combines face reconstruction and occlusion area recognition into the same model. When recognizing the occlusion area, the relevant data in the face reconstruction process is used to model the occlusion problem on the two-dimensional plane as the recognition of three-dimensional face occlusion, which is more in line with the way of human perception. At the same time, because the relevant data of the three-dimensional face is used, it is easier to mine hidden related features, thereby obtaining a more accurate occlusion area recognition result. At the same time, the common characteristics of face reconstruction and occluded area recognition tasks are fully utilized, and the two tasks are mutually reinforcing while compressing the model as much as possible, optimizing the overall resource deployment, and being able to meet application scenarios with many parallel tasks and high real-time requirements, and the accuracy of occluded area judgment is higher.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for face reconstruction and occlusion area recognition provided by an embodiment of the present application;

FIG2 is a flow chart of a method for performing three-dimensional reconstruction of a face image based on a reconstruction parameter vector provided in an embodiment of the present application;

FIG3 is a schematic diagram of a picture of reconstruction and processing of a face image provided by an embodiment of the present application;

FIG4 is a flow chart of a method for constructing graph structure information based on a depth feature map and two-dimensional face information provided by an embodiment of the present application;

FIG5 is a schematic diagram of performing special effects processing on an image provided by an embodiment of the present application;

FIG6 is a structural block diagram of a face reconstruction and occlusion area recognition device provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of the structure of a face reconstruction and occlusion area recognition device provided in an embodiment of the present application.

Detailed ways

The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only parts related to the embodiments of the present application are shown in the accompanying drawings, rather than all structures.

The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and/or" in the specification and claims represents at least one of the connected objects, and the character "/" generally indicates that the objects associated with each other are in an "or" relationship.

The face reconstruction and occlusion area recognition method provided in the embodiments of the present application can be applied to various scenarios that require 3D face reconstruction, and can also accurately identify whether the current face is occluded and the corresponding occlusion area. Specifically, the face image in the short video or live broadcast process is processed to achieve face reconstruction and occlusion area recognition. After the occlusion area recognition is completed, makeup and other special effects processing can be further performed based on the occlusion area recognition result to ensure the special effects.

FIG1 is a flow chart of a method for face reconstruction and occlusion area recognition provided by an embodiment of the present application, which specifically includes the following steps:

Step S101: input a face image into a first network structure, and output a reconstruction parameter vector of the face image when three-dimensional reconstruction is performed through the first network structure.

In one embodiment, the face image is an image containing a face area, and the face image may be an image captured by a camera or a picture input by a user. Optionally, before the face image is input into the first network structure, a face detector may be used to perform relevant face detection and alignment correction on the face image.

In one embodiment, the first network structure may be a convolutional neural network, such as a mobilenet-v3 network. The first network structure is a pre-trained network structure, which takes a two-dimensional image as input during training, and outputs a reconstruction parameter vector for three-dimensional face reconstruction through a series of convolutional layers. Optionally, the first network structure uses a reconstruction loss function of weakly supervised training during training.

In one embodiment, after the face image is input into the first network structure, the reconstruction parameter vector output by the first network structure exemplarily includes: a face feature vector, a face expression vector and a three-dimensional face coefficient vector. Among them, other vectors used for three-dimensional reconstruction are also included, such as an illumination vector, a reflectivity vector, a posture vector and a translation vector.

Step S102: Perform three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information.

In one embodiment, after obtaining the reconstruction parameter vector corresponding to the input face image through the first network structure, a three-dimensional reconstruction of the face image is performed based on the reconstruction parameter vector to obtain three-dimensional face information. An exemplary three-dimensional reconstruction process is shown in FIG2, which is a flow chart of a method for three-dimensional reconstruction of a face image based on a reconstruction parameter vector provided in an embodiment of the present application, specifically comprising:

Step S1021, reconstructing a three-dimensional face point cloud based on the face feature vector, the face expression vector, the three-dimensional face coefficient vector and the preset average face shape information to obtain point cloud reconstruction information.

In one embodiment, the three-dimensional face reconstruction includes the reconstruction of the three-dimensional face point cloud and the generation of face texture information. Optionally, during the reconstruction of the three-dimensional face point cloud, the reconstruction is performed based on the face feature vector, the face expression vector, the three-dimensional face coefficient vector in the determined reconstruction parameter vector, and the preset average face shape information.

For example, the facial feature vector is recorded as _Bid , the facial expression vector is recorded as _Bexp , the three-dimensional facial coefficient vector includes α, β and δ, and the preset average facial shape information is recorded as For example, when determining the point cloud reconstruction information S, the following formula is used to calculate:

Step S1022: reconstruct the facial texture based on the three-dimensional facial coefficient vector and the preset average facial texture information and facial base information to obtain facial texture information.

In one embodiment, when generating the facial texture information, based on the three-dimensional facial coefficient vector in the reconstruction parameter vector determined above, the acquired facial base information and the preset average facial texture Optionally, the face base information may use base information in a public face model.

For example, the three-dimensional face coefficient vector includes α, β and δ, the face base information is recorded as B _t , and the preset average face texture information is recorded as When determining the face texture information T, the following formula is used to calculate it:

Step S1023: Generate three-dimensional face information based on the point cloud reconstruction information and the face texture information.

In one embodiment, after obtaining the point cloud reconstruction information and the face texture information, the final three-dimensional face information is generated based on the point cloud reconstruction information and the face texture information. Exemplarily, the three-dimensional face information can be obtained by superimposing and fitting the point cloud reconstruction information and the face texture information.

Step S103: Render and map the three-dimensional face information to generate two-dimensional face information containing network segmentation data.

In one embodiment, after obtaining the three-dimensional face information, it is further rendered and mapped to generate two-dimensional face information, wherein the generated two-dimensional face information contains network segmentation data. Optionally, the rendering process may be to render the three-dimensional face information by a renderer to obtain a two-dimensional face image, or to render by using a set rendering model or other rendering algorithm; the mapping process may be: based on the topological structure in the three-dimensional face information, the three-dimensional face point pair data in the three-dimensional face information is mapped to the two-dimensional face image. Exemplarily, as shown in FIG3, FIG3 is a picture schematic diagram of the reconstruction and processing of the face image provided by the embodiment of the present application, wherein the original input face image is _IT , and the image obtained after three-dimensional reconstruction is _VR . The image _VR can be rendered in two dimensions using a differential renderer to obtain a two-dimensional face image _IR . At this time, the triangular face of 1293 three-dimensional face point pairs is projected into the two-dimensional face image _IR in combination with the topological structure of the reconstructed image _VR to generate two-dimensional face information _TrR containing network segmentation data.

Step S104: Obtain a depth feature map extracted by the first network structure, and construct graph structure information based on the depth feature map and the two-dimensional face information.

In one embodiment, when identifying the occluded area, the depth feature map extracted by the first network structure during the three-dimensional reconstruction process is used, and the graph structure information is constructed based on the depth feature map and the two-dimensional face information generated in step S103, and the occluded area is further identified based on the graph structure information. The depth feature map is the feature of the layer obtained after calculating the convolution layer in the first network structure.

Optionally, an example of a method of constructing graph structure information using the depth feature map and two-dimensional face information As shown in FIG. 4 , FIG. 4 is a flow chart of a method for constructing graph structure information based on a depth feature map and two-dimensional face information provided by an embodiment of the present application, specifically comprising:

Step S1041: Segment the depth feature map based on the network segmentation data in the two-dimensional face information to obtain a triangular face segmentation result.

When segmenting the depth feature map, the same segmentation strategy as the network segmentation in the two-dimensional face information is used for segmentation. When segmenting, the size of the depth feature map and the image size in the two-dimensional face information are further adjusted to be consistent in size so that the two sizes are the same. At this time, the same segmentation strategy is executed to obtain a triangular face segmentation result.

Step S1042: construct an adjacency matrix based on the vertex connectivity and point pair distances in the triangle face segmentation result to obtain graph structure information.

After obtaining the triangular face segmentation result, the adjacency matrix is constructed using the vertex connectivity of the three-dimensional face mesh itself and the point-to-point distance between the point pairs after projection onto the two-dimensional plane, thereby completing the construction of the graph structure information. For example, taking the construction of the adjacency matrix A _ij ∈ NxN as an example, where N represents the number of vertices of the three-dimensional face point cloud, and in one embodiment, 1293 vertices are used. The specific method of constructing the adjacency matrix based on the vertex connectivity and the point-to-point distance is:
A _ij =C _ij *D _ij

Among them, C _ij represents whether two vertices are connected. If they are not connected, the value is 0, otherwise the value is 1. _Dij represents the point pair distance between the two vertices after projection onto the two-dimensional plane.

Step S105: input the graph structure information into a second network structure, and output the occluded area corresponding to the face image through the second network structure.

In one embodiment, the second network structure may be a graph convolutional neural network, and the graph structure information generated in step S104 is input into the second network structure, and the occlusion area corresponding to the face image is output through the second network structure. Optionally, the three-dimensional face vertices in the graph structure information may be classified through a graph convolutional neural network, and the occlusion area corresponding to the face image is output based on the classification result. Optionally, the framework of the second network structure mainly uses a graph attention network, which integrates and classifies the three-dimensional face vertices and finally outputs a segmentation mask _MR of the occlusion area.

Optionally, the second network structure uses a supervised segmentation loss function during training, such as using a dice loss function to supervise the occlusion mask and the true mask. Experiments have shown that it is effective for segmentation. The prediction of mask can obtain more accurate results compared with the traditional cross entropy loss function.

It can be seen from the above scheme that by inputting a face image into the first network structure, the reconstruction parameter vector of the face image during three-dimensional reconstruction is output through the first network structure, and then the face image is reconstructed in three dimensions based on the reconstruction parameter vector to generate three-dimensional face information, and the three-dimensional face information is rendered and mapped to generate two-dimensional face information containing network segmentation data. At this time, the depth feature map extracted by the first network structure is further obtained, and the graph structure information is constructed based on the depth feature map and the two-dimensional face information. The graph structure information is input into the second network structure, and the occlusion area corresponding to the face image is output through the second network structure. The face reconstruction and occlusion area recognition method provided by this scheme combines face reconstruction and occlusion area recognition into the same model. When recognizing the occlusion area, the relevant data in the face reconstruction process is used to model the occlusion problem on the two-dimensional plane as the recognition of three-dimensional face occlusion, which is more in line with the way of human perception. At the same time, because the relevant data of the three-dimensional face is used, it is easier to mine hidden related features, thereby obtaining a more accurate occlusion area recognition result. At the same time, the common characteristics of face reconstruction and occluded area recognition tasks are fully utilized, and the two tasks are mutually reinforcing while compressing the model as much as possible, optimizing the overall resource deployment, and being able to meet application scenarios with many parallel tasks and high real-time requirements, and the accuracy of occluded area judgment is higher.

In the above solution, face reconstruction and occluded area recognition are integrated into one model for processing, and feature extraction is performed on the two related tasks at the same time, thereby solving the pain point that traditional solutions cannot mine the hidden related features of the two tasks, compressing the model size, and achieving better reconstruction and segmentation effects.

On the basis of the above scheme, after the occlusion area corresponding to the face image is output through the second network structure, it also includes: performing special effect rendering processing based on the face image, two-dimensional face information and occlusion area, and displaying the processing results. That is, after three-dimensional reconstruction and determination of the occlusion area, various special effects processing can be further applied based on this. Exemplarily, as shown in Figure 5, Figure 5 is a schematic diagram of special effect processing of an image provided by an embodiment of the present application. Among them, the face image _IT is the original input image, the image _MR is the image output after the occlusion area is identified, and the image _IR is the image corresponding to the two-dimensional face information after rendering and mapping. The three are processed to finally obtain the image I _F. Optionally, in the process of special effect rendering processing, the occlusion area can be rendered using the original image occlusion object, and the part of the face image outside the occlusion area can be rendered based on the three-dimensional face information.

The black part in the image _MR represents the unobstructed area, indicating that the original image has not been covered in this area. Occlusion, the area should be rendered into the reconstructed image _IR during the final rendering. Conversely, the white area in the image _MR represents the area of the original input image that is occluded, and the reconstructed image should not be rendered during the final rendering. Since a complete face will be reconstructed regardless of whether the object face is occluded during three-dimensional face reconstruction. Therefore, for situations such as hand self-occlusion and sunglasses occlusion in the example of Figure 5, only the entire face will be reconstructed as shown in the image _IR during reconstruction, and no occluders will be reconstructed. If there is a post-processing process similar to three-dimensional makeup, since a complete three-dimensional face is reconstructed, the entire three-dimensional makeup material will be retained, and the occluded part of the makeup will not be shielded due to the presence of occluders. Therefore, in order to solve this common problem, in one embodiment, the occluder is identified using the predicted black area of the image _MR , and the input image is finally rendered for the occluded part. The parts blocked by the sunglasses and hands are rendered using the sunglasses and hands in the image _IT in the final rendered image I _F , thereby avoiding the problem of being unable to render the blocked objects or, if there is three-dimensional makeup post-processing, mistakenly rendering the three-dimensional makeup onto the blocked objects.

FIG6 is a structural block diagram of a face reconstruction and occlusion region identification device provided in an embodiment of the present application. The device is used to execute the face reconstruction and occlusion region identification method provided in the above embodiment, and has the corresponding functional modules and beneficial effects of the execution method. As shown in FIG6, the device specifically includes: a parameter vector generation module 101, a three-dimensional information determination module 102, a rendering mapping module 103, a graph information construction module 104 and an occlusion region determination module 105, wherein:

The parameter vector generation module 101 is configured to input a face image into a first network structure, and output a reconstruction parameter vector of the face image when performing three-dimensional reconstruction through the first network structure;

A three-dimensional information determination module 102, configured to perform three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information;

A rendering and mapping module 103 is configured to render and map the three-dimensional face information to generate two-dimensional face information including network segmentation data;

A graph information construction module 104, configured to obtain a depth feature graph extracted by the first network structure, and construct graph structure information based on the depth feature graph and the two-dimensional face information;

The occlusion region determination module 105 is configured to input the graph structure information into a second network structure, and output the occlusion region corresponding to the face image through the second network structure.

It can be seen from the above scheme that the face image is input into the first network structure, the reconstruction parameter vector of the face image is output through the first network structure when performing three-dimensional reconstruction, and then the face is reconstructed based on the reconstruction parameter vector. The three-dimensional reconstruction of the image generates three-dimensional face information, and the three-dimensional face information is rendered and mapped to generate two-dimensional face information containing network segmentation data. At this time, the depth feature map extracted by the first network structure is further obtained, and the graph structure information is constructed based on the depth feature map and the two-dimensional face information. The graph structure information is input into the second network structure, and the occlusion area corresponding to the face image is output through the second network structure. The face reconstruction and occlusion area recognition method provided by this scheme merges face reconstruction and occlusion area recognition into the same model. When recognizing the occlusion area, the relevant data in the face reconstruction process is used to model the occlusion problem on the two-dimensional plane as the recognition of three-dimensional face occlusion, which is more in line with the way of human perception. At the same time, due to the use of the relevant data of the three-dimensional face, it is easier to mine the hidden related features, so as to obtain a more accurate occlusion area recognition result. At the same time, the common characteristics of the two tasks of face reconstruction and occlusion area recognition are fully utilized, and the two tasks are mutually promoted while compressing the model as much as possible, optimizing the overall resource deployment, and being able to meet the application scenarios with many parallel tasks and high real-time requirements, and the accuracy of occlusion area determination is higher.

In a possible embodiment, the reconstruction parameter vector includes a face feature vector, a face expression vector and a three-dimensional face coefficient vector, and the three-dimensional information determination module 102 is configured as follows:

Reconstructing a three-dimensional face point cloud based on the face feature vector, the face expression vector, the three-dimensional face coefficient vector and preset average face shape information to obtain point cloud reconstruction information;

Reconstructing facial texture based on the three-dimensional facial coefficient vector and preset average facial texture information and facial base information to obtain facial texture information;

Generate three-dimensional face information based on the point cloud reconstruction information and the face texture information.

In a possible embodiment, the rendering mapping module 103 is configured as follows:

Rendering the three-dimensional face information by a renderer to obtain a two-dimensional face image;

Based on the topological structure in the three-dimensional face information, the three-dimensional face point pair data in the three-dimensional face information is mapped to the two-dimensional face image to generate two-dimensional face information containing network segmentation data.

In a possible embodiment, the graph information construction module 104 is configured as follows:

Segment the depth feature map based on the network segmentation data in the two-dimensional face information to obtain a triangular face segmentation result;

The graph structure information is obtained by constructing an adjacency matrix based on the vertex connectivity and point pair distances in the triangle face segmentation result.

In a possible embodiment, the occlusion area determination module 105 is configured as follows:

Classifying the three-dimensional face vertices in the graph structure information by using the graph convolutional neural network;

The occluded area corresponding to the face image is output based on the classification result.

In a possible embodiment, the device further includes a special effect processing module configured to:

After the occluded area corresponding to the face image is output through the second network structure, special effects rendering processing is performed based on the face image, the two-dimensional face information and the occluded area, and the processing result is displayed.

In a possible embodiment, the special effect processing module is configured as follows:

The occluded area is rendered using the original image occluder, and the facial image portion outside the occluded area is rendered based on the three-dimensional facial information.

FIG7 is a schematic diagram of the structure of a face reconstruction and occlusion area recognition device provided by an embodiment of the present application. As shown in FIG7, the device includes a processor 201, a memory 202, an input device 203, and an output device 204; the number of processors 201 in the device can be one or more, and FIG7 takes one processor 201 as an example; the processor 201, the memory 202, the input device 203, and the output device 204 in the device can be connected by a bus or other means, and FIG7 takes the connection by a bus as an example. The memory 202, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and modules, such as program instructions/modules corresponding to the face reconstruction and occlusion area recognition method in the embodiment of the present application. The processor 201 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 302, that is, realizing the above-mentioned face reconstruction and occlusion area recognition method. The input device 203 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the device. The output device 204 may include a display device such as a display screen.

The embodiment of the present application further provides a non-volatile storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute a face reconstruction and occlusion area recognition method described in the above embodiment when executed by a computer processor, including:

Inputting a face image into a first network structure, and outputting a reconstruction parameter vector of the face image when three-dimensional reconstruction is performed through the first network structure;

Performing three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information;

Rendering and mapping the three-dimensional face information to generate two-dimensional face information containing network segmentation data;

Obtain a depth feature map extracted by the first network structure, based on the depth feature map and the Two-dimensional face information construction graph structure information;

The graph structure information is input into a second network structure, and the occluded area corresponding to the face image is output through the second network structure.

It is worth noting that in the above-mentioned embodiment of the face reconstruction and occlusion area recognition device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific name of each functional unit is only for the convenience of distinguishing each other, and is not used to limit the protection scope of the embodiment of the present application.

In some possible implementations, various aspects of the method provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the face reconstruction and occlusion area recognition method recorded in the embodiment of the present application. The program product may be implemented in any combination of one or more readable media.

Claims (11)

A method for face reconstruction and occlusion region recognition, comprising: Inputting a face image into a first network structure, and outputting a reconstruction parameter vector of the face image when three-dimensional reconstruction is performed through the first network structure; Performing three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information; Rendering and mapping the three-dimensional face information to generate two-dimensional face information containing network segmentation data; Obtaining a depth feature map extracted by the first network structure, and constructing graph structure information based on the depth feature map and the two-dimensional face information; The graph structure information is input into a second network structure, and the occluded area corresponding to the face image is output through the second network structure. The method for face reconstruction and occlusion area recognition according to claim 1, wherein the reconstruction parameter vector includes a face feature vector, a face expression vector and a three-dimensional face coefficient vector, and the three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information comprises: Reconstructing a three-dimensional face point cloud based on the face feature vector, the face expression vector, the three-dimensional face coefficient vector and preset average face shape information to obtain point cloud reconstruction information; Reconstructing facial texture based on the three-dimensional facial coefficient vector and preset average facial texture information and facial base information to obtain facial texture information; Generate three-dimensional face information based on the point cloud reconstruction information and the face texture information. The method for face reconstruction and occlusion area recognition according to claim 1, wherein the rendering and mapping of the three-dimensional face information to generate two-dimensional face information containing network segmentation data comprises: Rendering the three-dimensional face information by a renderer to obtain a two-dimensional face image; Based on the topological structure in the three-dimensional face information, the three-dimensional face point pair data in the three-dimensional face information is mapped to the two-dimensional face image to generate two-dimensional face information containing network segmentation data. The method for face reconstruction and occlusion area recognition according to any one of claims 1 to 3, wherein the constructing graph structure information based on the depth feature map and the two-dimensional face information comprises: Segment the depth feature map based on the network segmentation data in the two-dimensional face information to obtain a triangular face segmentation result; The adjacency matrix is constructed based on the vertex connectivity and point pair distance in the triangular face segmentation result. The graph structure information is obtained by constructing . The method for face reconstruction and occlusion area recognition according to any one of claims 1 to 4, wherein the second network structure comprises a graph convolutional neural network, and outputting the occlusion area corresponding to the face image through the second network structure comprises: Classifying the three-dimensional face vertices in the graph structure information by using the graph convolutional neural network; The occluded area corresponding to the face image is output based on the classification result. The method for face reconstruction and occlusion area recognition according to any one of claims 1 to 5, wherein, after outputting the occlusion area corresponding to the face image through the second network structure, it also includes: Special effects rendering processing is performed based on the facial image, the two-dimensional facial information and the occluded area, and the processing result is displayed. The method for face reconstruction and occlusion area recognition according to claim 6, wherein the special effects rendering processing based on the face image, the two-dimensional face information and the occlusion area comprises: The occluded area is rendered using the original image occluder, and the facial image portion outside the occluded area is rendered based on the three-dimensional facial information. A face reconstruction and occlusion area recognition device, comprising: a parameter vector generation module, configured to input a face image into a first network structure, and output a reconstruction parameter vector of the face image when three-dimensional reconstruction is performed through the first network structure; a three-dimensional information determination module, configured to perform three-dimensional reconstruction of the face image based on the reconstruction parameter vector to generate three-dimensional face information; A rendering and mapping module, configured to render and map the three-dimensional face information to generate two-dimensional face information containing network segmentation data; A graph information construction module, configured to obtain a deep feature map extracted by the first network structure, and construct graph structure information based on the deep feature map and the two-dimensional face information; The occlusion area determination module is configured to input the graph structure information into a second network structure, and output the occlusion area corresponding to the face image through the second network structure. A face reconstruction and occlusion area recognition device, the device comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the face reconstruction and occlusion area recognition method described in any one of claims 1-7. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the face reconstruction and occlusion area recognition method according to any one of claims 1 to 7 when executed by a computer processor. A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for face reconstruction and occlusion area recognition according to any one of claims 1 to 7 is implemented.