JP2024522287A - 3D human body reconstruction method, apparatus, device and storage medium - Google Patents

Abstract

The embodiment of the present invention provides a 3D human body reconstruction method, apparatus, device and storage medium. The method may include: performing human body geometric reconstruction according to a human body image of a target human body to obtain a 3D mesh model of the target human body; performing local geometric reconstruction on a local part of the target human body according to the human body image to obtain a 3D mesh model of the local part; fusing the 3D mesh model of the local part and the 3D mesh model of the target human body to obtain an initial 3D model; and performing human body texture reconstruction according to the initial 3D model and the human body image to obtain a 3D human body model of the target human body. According to the embodiment of the present invention, the local part in the 3D mesh model of the target human body is more clear and accurate, and the reconstruction effect of the local part is improved.

Description

[Cross-reference to related applications]
This application claims priority to a Chinese patent application filed on March 31, 2021, with application number 202110352199.4 and entitled "Three-dimensional human body reconstruction method, apparatus, device and storage medium," the contents of which are incorporated herein by reference.

The present invention relates to image processing technology, and more specifically to a method, apparatus, device and storage medium for three-dimensional human body reconstruction.

3D human body reconstruction is an important problem in the fields of computer vision and computer graphics. The reconstructed human body digital model has important applications in many fields, such as anthropometry, virtual fitting, virtual live streamers, custom design of game characters, and virtual reality social. Among them, how to project the real-world human body into the virtual world to obtain a 3D human body digital model has become an important issue. However, the digital reconstruction of the 3D human body is very complicated, and the scanner needs to continuously scan the object at multiple angles without blind spots, and the reconstruction result also has the problem that the local reconstruction effect is not sensitive enough.

In view of this, embodiments of the present invention provide at least a method, apparatus, device, and storage medium for three-dimensional human body reconstruction.

A first aspect provides a method for reconstructing a three-dimensional human body, the method comprising:
performing human body geometry reconstruction based on the human body image of the target human body to obtain a 3D mesh model of the target human body;
performing local geometric reconstruction on a local portion of the target human body based on a human body image of the target human body to obtain a three-dimensional mesh model of the local portion;
Fusing the 3D mesh model of the local region with the 3D mesh model of the target human body to obtain an initial 3D model;
and reconstructing a body texture of the target human body based on the initial 3D model and the human body image to obtain a 3D human body model of the target human body.

In one example, the step of performing human body geometric reconstruction based on the human body image of the target human body and obtaining a three-dimensional mesh model of the target human body includes the steps of performing three-dimensional reconstruction on the human body image of the target human body via a first deep neural network branch to obtain a first human body model, performing three-dimensional reconstruction on a local image in the human body image via a second deep neural network branch to obtain a second human body model, fusing the first human body model and the second human body model to obtain a fused human body model, and performing a meshing process on the fused human body model to obtain a three-dimensional mesh model of the target human body, where the local image includes a local region of the target human body.

In one example, the first deep neural network branch includes a global feature sub-network and a first fitting sub-network, and the second deep neural network branch includes a local feature sub-network and a second fitting sub-network. The step of performing three-dimensional reconstruction on the human body image of the target human body via the first deep neural network branch and obtaining a first human body model includes a step of performing feature extraction on the human body image via the global feature sub-network and obtaining a first image feature, and a step of obtaining the first human body model based on the first image feature via the first fitting sub-network. The step of performing three-dimensional reconstruction on a local image in the human body image via the second deep neural network branch and obtaining a second human body model includes a step of performing feature extraction on the local image via the local feature sub-network and obtaining a second image feature, and a step of obtaining the second human body model via the second fitting sub-network based on the second image feature and the intermediate feature output from the first fitting sub-network.

In one example, the step of performing local geometric reconstruction on a local portion of the target human body based on a human body image of the target human body and obtaining a three-dimensional mesh model of the local portion includes a step of performing feature extraction on the human body image of the target human body and obtaining a third image feature, and a step of identifying a three-dimensional mesh model of the local portion based on the third image feature and a three-dimensional topology template of the local portion.

In one example, the step of fusing the 3D mesh model of the local region with the 3D mesh model of the target human body to obtain the initial 3D model includes the steps of: acquiring a plurality of key points of the local region based on a human body image of the target human body; identifying information of first model key points corresponding to the plurality of key points in the 3D mesh model of the target human body, and identifying information of second model key points corresponding to the plurality of key points in the 3D mesh model of the local region; and fusing the 3D mesh model of the local region with the 3D mesh model of the target human body based on the information of the first model key points and the information of the second model key points to obtain the initial 3D model.

In one example, the step of fusing the 3D mesh model of the local region with the 3D mesh model of the target human body based on the information of the first model keypoints and the information of the second model keypoints to obtain the initial 3D model includes the steps of: determining a coordinate transformation relationship between the 3D mesh model of the target human body and the 3D mesh model of the local region based on the information of the first model keypoints and the information of the second model keypoints; transforming the 3D mesh model of the local region into the coordinate system of the 3D mesh model of the target human body based on the coordinate transformation relationship; and fusing the 3D mesh model of the local region with the 3D mesh model of the target human body in the transformed coordinate system to obtain the initial 3D model.

In one example, the human body image includes a front texture of the target human body and a background image, and the step of reconstructing the human body texture of the target human body based on the initial three-dimensional model and the human body image and obtaining a three-dimensional human body model of the target human body includes a step of performing human body segmentation on the human body image to obtain a first segmentation mask, a second segmentation mask, and a front texture of the target human body, a step of inputting the front texture, the first segmentation mask, and the second segmentation mask into a texture generation network to obtain a back texture of the target human body, and a step of obtaining a three-dimensional human body model having texture corresponding to the target human body based on the back texture and the front texture, where the first segmentation mask corresponds to a mask region of the front texture, and the second segmentation mask corresponds to a mask region of the back texture of the target human body.

In one example, training of the texture generation network includes a process of performing human body segmentation on an image of a human body sample in a training sample image set to obtain a first sample segmentation mask, a second sample segmentation mask, and a frontal texture of the human body sample, a process of training the supporting texture generation network based on the frontal texture of the human body in the supporting human body image obtained by reducing the resolution of the image of the human body sample, a third sample segmentation mask, and a fourth sample segmentation mask, and after the training of the supporting texture generation network is completed, a process of performing human body segmentation on an image of the human body sample in a training sample image set to obtain a first sample segmentation mask, a second sample segmentation mask, and a frontal texture of the human body sample. and a process of training the texture generation network based on a sample segmentation mask, the first sample segmentation mask corresponding to a mask region of the front texture of the human body sample, the second sample segmentation mask corresponding to a mask region of the back texture of the human body sample, the third sample segmentation mask corresponding to a mask region of the front texture of the human body in the supporting human body image, and the fourth sample segmentation mask corresponding to a mask region of the back texture of the human body in the supporting human body image, and the network parameters of the texture generation network include at least a portion of the network parameters of the supporting texture generation network that has completed training.

In one example, the localized portion of the target human body is the face of the target human body, and/or the human body image is an RGB image.

In one example, the 3D human body reconstruction method further includes a step of acquiring a human body skeletal structure of the target human body when performing human body geometric reconstruction based on a human body image of the target human body, and a step of determining skinning weights for driving the 3D human body model based on the 3D human body model and the human body skeletal structure after the 3D human body model of the target human body is acquired.

A second aspect provides a three-dimensional human body reconstruction apparatus, comprising:
a global reconstruction module for performing human body geometric reconstruction based on the human body image of a target human body to obtain a three-dimensional mesh model of the target human body;
a local reconstruction module for performing local geometric reconstruction on a local portion of the target human body based on a human body image of the target human body to obtain a three-dimensional mesh model of the local portion;
a fusion processing module for fusing the 3D mesh model of the local region and the 3D mesh model of the target human body to obtain an initial 3D model;
a texture reconstruction module for reconstructing a body texture of the target human body based on the initial 3D model and the human body image to obtain a 3D human body model of the target human body.

In one example, when obtaining a three-dimensional mesh model of the target human body, the overall reconstruction module specifically performs three-dimensional reconstruction on a human body image of the target human body via a first deep neural network branch to obtain a first human body model, performs three-dimensional reconstruction on a local image in the human body image via a second deep neural network branch to obtain a second human body model, fuses the first human body model and the second human body model to obtain a fused human body model, and performs a meshing process on the fused human body model to obtain a three-dimensional mesh model of the target human body, and the local image includes a local region of the target human body.

In one example, the local reconstruction module is specifically used to perform feature extraction on the human body image of the target human body, obtain third image features, and identify a three-dimensional mesh model of the local region based on the third image features and a three-dimensional topology template of the local region.

In one example, the fusion processing module is specifically used to obtain a plurality of key points of the local region based on a human body image of the target human body, identify information of first model key points corresponding to the plurality of key points in a three-dimensional mesh model of the target human body, and identify information of second model key points corresponding to the plurality of key points in the three-dimensional mesh model of the local region, and fuse the three-dimensional mesh model of the local region with the three-dimensional mesh model of the target human body based on the information of the first model key points and the information of the second model key points to obtain the initial three-dimensional model.

In one example, the fusion processing module is used to fuse the 3D mesh model of the local region with the 3D mesh model of the target human body based on the information of the first model keypoints and the information of the second model keypoints to obtain the initial 3D model, specifically, to determine a coordinate transformation relationship between the 3D mesh model of the target human body and the 3D mesh model of the local region based on the information of the first model keypoints and the information of the second model keypoints, transform the 3D mesh model of the local region into the coordinate system of the 3D mesh model of the target human body based on the coordinate transformation relationship, fuse the 3D mesh model of the local region with the 3D mesh model of the target human body in the transformed coordinate system, and obtain the initial 3D model.

In one example, the texture reconstruction module specifically performs body segmentation on the human body image, obtains a first segmentation mask, a second segmentation mask, and a front texture of the target human body, inputs the front texture, the first segmentation mask, and the second segmentation mask into a texture generation network to obtain a back texture of the target human body, and is used to obtain a three-dimensional human body model having texture corresponding to the target human body based on the back texture and the front texture, where the first segmentation mask corresponds to a mask area of the front texture and the second segmentation mask corresponds to a mask area of the back texture of the target human body.

In one example, the 3D human body reconstruction device further includes a model training module for training the texture generation network, and the model training module specifically performs human body segmentation on an image of a human body sample in a training sample image set, obtains a first sample segmentation mask, a second sample segmentation mask, and a front texture of the human body sample, and trains an assisting texture generation network based on the front texture of the human body in the assisting human body image obtained by reducing the resolution of the image of the human body sample, a third sample segmentation mask, and a fourth sample segmentation mask, and after the training of the assisting texture generation network is completed, The texture generation network is trained based on the model, the first sample division mask, and the second sample division mask, the first sample division mask corresponds to a mask region of the front texture of the human body sample, the second sample division mask corresponds to a mask region of the back texture of the human body sample, the third sample division mask corresponds to a mask region of the front texture of the human body in the supporting human body image, and the fourth sample division mask corresponds to a mask region of the back texture of the human body in the supporting human body image, and the network parameters of the texture generation network include at least a portion of the network parameters of the supporting texture generation network that has completed training.

A third aspect provides an electronic device, comprising a memory and a processor, the memory adapted to store computer-readable instructions, and the processor adapted to perform a method according to any of the embodiments of the present invention by invoking the computer instructions.

A fourth aspect provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a method according to any one of the embodiments of the present invention is performed.

A fifth aspect provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs a method according to any of the embodiments of the present invention.

In the three-dimensional human body reconstruction method, apparatus, device, and storage medium according to the embodiments of the present invention, local geometric reconstruction is performed on a local portion of a target human body, and a three-dimensional mesh model of the local portion obtained by the local geometric reconstruction is merged with a three-dimensional mesh model of the target human body, so that the local portion in the three-dimensional mesh model of the target human body becomes clearer, more delicate, and more accurate, and the reconstruction effect of the local portion is improved. In addition, the method can perform reconstruction based on a single human body image of the target human body, which simplifies the user's cooperation procedure and makes three-dimensional human body reconstruction easier.

In order to make the technical solutions in one or more embodiments of the present invention or related art more clearly described, the following briefly introduces drawings necessary for describing the embodiments of the present invention or related art. Obviously, the drawings in the following description are merely some embodiments described in one or more embodiments of the present invention, and those skilled in the art can obtain other drawings from these drawings without creative efforts.
1 shows a flowchart of a method for 3D body reconstruction in accordance with at least one embodiment of the present invention. 1 illustrates a schematic diagram of a scheme for obtaining a 3D mesh model based on a single human body image in accordance with at least one embodiment of the present invention; 1 shows a schematic diagram of a procedure for obtaining an initial 3D model in accordance with at least one embodiment of the present invention; 1 shows a schematic diagram of a texture reconstruction procedure in accordance with at least one embodiment of the present invention; 1 shows a schematic diagram of a procedure for determining skinning weights in accordance with at least one embodiment of the present invention; 1 illustrates a schematic diagram of a scheme for obtaining a 3D mesh model based on a single human body image in accordance with at least one embodiment of the present invention; FIG. 2 shows a principle schematic diagram of texture generation according to at least one embodiment of the present invention. FIG. 1 shows a schematic diagram of a training procedure for a texture generating network in accordance with at least one embodiment of the present invention. 1 shows a schematic diagram of a human body image in accordance with at least one embodiment of the present invention; FIG. 1 shows a block diagram of a 3D body reconstruction device in accordance with at least one embodiment of the present invention. FIG. 1 shows a block diagram of a 3D body reconstruction device in accordance with at least one embodiment of the present invention.

In order to make the technical solutions in one or more embodiments of the present invention better understood by those skilled in the art, the technical solutions in one or more embodiments of the present invention are described below clearly and completely in combination with drawings in one or more embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art without creative efforts based on one or more embodiments of the present invention should be included in the protection scope of the present invention.

3D human body reconstruction has important applications in many fields, including but not limited to the following application scenarios:

For example, 3D human body reconstruction can enhance the realism of some virtual reality application scenarios, such as virtual fitting, virtual cloud meetings, virtual classes, etc.

Furthermore, for example, a 3D human body model obtained by 3D human body reconstruction may be introduced into game data to create a personalized human character.

Furthermore, for example, currently, the creation of science fiction movies requires the use of various scientific technologies such as green screen, motion capture, etc., the hardware devices are expensive, and the overall flow is time-consuming and cumbersome. By obtaining a virtual 3D human body model through 3D human body reconstruction, the flow can be simplified and resources can be saved.

Regardless of the application scenario, there are the following demands for 3D human body reconstruction. On the one hand, it is necessary to simplify the user cooperation procedure as much as possible, for example, the user needs to cooperate with multi-angle scanning, which forces the user to provide a lot of cooperation, resulting in a poor experience for the user. On the other hand, it is necessary to obtain a 3D human body model with higher accuracy as much as possible, for example, in a virtual cloud meeting or AR virtual interaction scene, the 3D human body model obtained by 3D human body reconstruction has a demand for higher realism and immersion.

To solve the above problems, an embodiment of the present invention provides a 3D human body reconstruction method. The method is based on one of the user's photos to perform 3D human body reconstruction for the user, simplifying the user's collaboration process and achieving a high-precision reconstruction effect.

As shown in FIG. 1, FIG. 1 illustrates a flowchart of a 3D body reconstruction method according to at least one embodiment of the present invention. The method may include steps 100 to 106.

In step 100, a human body geometry reconstruction is performed based on a single human body image of the target human body to obtain a three-dimensional mesh model of the target human body.

The target human body is the base user of the 3D human body reconstruction. For example, 3D human body reconstruction is performed for a user, Mr. Zhang, who may be called the target human body. The reconstructed 3D human body model is also obtained based on Mr. Zhang's body, and has a high similarity to Mr. Zhang's posture, appearance, clothing, hairstyle, etc.

The single human body image is a single human body image of the target human body. In the embodiment of the present invention, there is no special requirement regarding the collection method and format of the human body image. In one exemplary method, the single human body image may be a single whole-body front photograph of the target human body. Furthermore, for example, the single human body image may be an RGB color image. The acquisition cost of such an RGB format image is low. For example, when collecting images, there is no need to use a costly device such as a depth-of-field camera, and they can be collected with a normal photographing device.

In this step, a human body geometry reconstruction may be performed based on a single human body image of the target human body to obtain a 3D mesh model. The 3D mesh model is a 3D mesh that represents the human body geometry, and the mesh includes some vertices and faces.

In one example, this embodiment may further perform posture and body shape alignment fitting for the three-dimensional mesh Mesh obtained by the reconstruction and one parameterized human body model stored in advance. Specifically, the parameterized human body model includes one human body surface mesh and one group of skeletal structures, which are controlled by one group of posture and body shape parameters, and the human body skeletal position and surface shape change with changes in parameter values. After performing geometric alignment for the three-dimensional mesh Mesh obtained by the reconstruction in step 100 and the parameterized human body model, a skeletal structure corresponding to the three-dimensional mesh Mesh obtained by the reconstruction in step 100 is obtained. The skeletal structure is used to calculate skinning weights in a later step.

FIG. 2 illustrates a method of obtaining a 3D mesh model based on the reconstruction of a single human body image. As shown in FIG. 2, a single human body image 21 of a target human body may be input to a first deep neural network branch 22 for 3D reconstruction. In one exemplary embodiment, the first deep neural network branch 22 may include a global feature sub-network 221 and a first fitting sub-network 222.

Feature extraction may be performed on the single human body image 21 via the global feature sub-network 221 to obtain high-level image features of the single human body image 21. The high-level image features may be referred to as first image features. For example, the global feature sub-network 221 may be a HourGlass convolutional network. The first image features are input to a first fitting sub-network 222. The first fitting sub-network 222 may predict whether each voxel block in the three-dimensional space belongs to the interior of the target human body based on the first image features. For example, the first fitting sub-network 222 may be a multi-layer perceptron structure. The first fitting sub-network 222 outputs a first human body model, which includes each three-dimensional voxel block located inside the target human body.

Next, a meshing process may be performed on the first human body model. For example, the meshing process may involve using a MarchingCubes algorithm in voxel space for the first human body model to obtain a three-dimensional mesh model of the target human body.

In step 102, local high-definition geometric reconstruction is performed on a local portion of the target body based on a single body image of the target body, and a three-dimensional mesh model of the local portion is obtained.

The 3D mesh model of the target body obtained by the reconstruction in step 100 may be blurred in localized areas of the target body. For example, the localized areas may be the face or other localized areas, such as the hands, that require detailed features to be embodied. Because the 3D mesh model is blurred in the facial details of the target body, and the face is typically an area of interest to users, in this step, geometric reconstruction may be performed individually for the localized areas of the target body.

Take the local part as an example, the face. The reconstruction of the human face may be a fine reconstruction of a fixed topology, that is, fitting may be performed on the positions of each vertex in the three-dimensional topology template of the face based on the image features obtained by performing feature extraction on a single human body image of the target human body, to obtain a three-dimensional mesh model of the face. Specifically, since the semantic structure of the human face has consistency, a three-dimensional face having a fixed topology structure may be adopted as a template. The template may be referred to as a three-dimensional topology template of the face. The template has multiple vertices, each of which corresponds to a fixed face meaning, for example, one vertex represents the tip of the nose and another vertex represents the corner of the eye. During face reconstruction, the positions of each vertex of the three-dimensional topology template of the face may be obtained by regression through a deep neural network.

For example, the deep neural network may include one deep convolutional network and one graph convolutional network. A single human body image of the target human body may be input to the deep convolutional network to extract image features. The extracted features may be referred to as third image features. Furthermore, the third image features and the three-dimensional topology template of the face may be input to the graph convolutional network to finally obtain a three-dimensional mesh model of the face output from the graph convolutional network. The three-dimensional mesh model is close to the face of the target human body. Optionally, the input of the deep convolutional network may be a portion of an image region including a face, which is cut out from the single human body image of the target human body.

In step 104, the 3D mesh model of the local area is fused with the 3D mesh model of the target human body to obtain an initial 3D model.

The 3D mesh model of the target human body obtained by the reconstruction in step 100 may be somewhat blurred at local parts of the human body. Take the local part as an example, the face. In step 102, the 3D mesh model of the face is obtained by individual geometric reconstruction of the face. In this step, the 3D mesh model of the face may replace the corresponding part in the 3D mesh model of the target human body in step 100. In this way, the five senses structure of the face can be made more delicate and accurate while retaining information such as head type, body type, and posture in the 3D mesh model of the target human body, thereby achieving a better reconstruction effect. Of course, for the sake of understanding, the local part is merely taken as an example of the face here, but in actual implementation, other local parts may be individually reconstructed to be clearer.

Specifically, a single human body image of a target human body may be input into a pre-trained keypoint detection model, and multiple keypoints of local parts of the target human body in the image may be identified through the keypoint detection model. Referring to FIG. 3, taking the local part as a face as an example, after multiple keypoints 31 of the face are obtained, corresponding model keypoints of the keypoints in the 3D mesh model of the target human body and the 3D mesh model of the face may be identified based on the coordinates of these keypoints 31 on the face. Specifically, information of multiple first model keypoints corresponding to multiple keypoints of the face in the 3D mesh model of the target human body may be identified. For example, the information may include a keypoint identifier of each first model keypoint and a corresponding keypoint position. Furthermore, information of second model keypoints corresponding to multiple keypoints of the face in the 3D mesh model of the face may be identified. For example, the information may include a keypoint identifier of each second model keypoint and a corresponding keypoint position.

After the first model keypoint information and the second model keypoint information are obtained, the 3D mesh model of the face may be fused with the 3D mesh model of the target body based on the first model keypoint information and the second model keypoint information to obtain an initial 3D model.

In an embodiment of the present invention, the step of fusing the 3D mesh model of the face with the 3D mesh model of the target human body may include the steps of: determining a coordinate transformation relationship between the 3D mesh model of the target human body and the 3D mesh model of the face using the camera external parameters of the two models based on the information of the first model key points and the information of the second model key points; transforming the 3D mesh model of the face into the coordinate system of the 3D mesh model of the target human body based on the coordinate transformation relationship; and fusing the 3D mesh model of the face with the 3D mesh model of the target human body in the transformed coordinate system. For example, the facial geometric structure on the 3D mesh model of the target human body may be removed and complemented with the 3D mesh model of the face, and the 3D mesh model of the face and the 3D mesh model of the target human body may be fused as a whole by the Poisson reconstruction method. The obtained model may be referred to as an initial 3D model. The initial 3D model has clear five sense structures and information such as similar head shape and posture, and has high accuracy.

In step 106, the body texture of the target body is reconstructed based on the initial 3D model and the single body image to obtain a 3D body model of the target body having a color texture.

In this embodiment, since 3D human body reconstruction is performed based on a single human body image of the target human body, some human body regions are invisible. For example, when reconstruction is performed using a front human body image of the target human body, the back side of the target human body is invisible, which causes a problem of missing texture. Therefore, in this step, based on the initial 3D model and the single human body image of the target human body, the human body texture of the invisible region of the target human body is predicted and complemented, and then merged with the human body texture in the single human body image to generate a texture-complete 3D human body model.

As shown in FIG. 4, for example, if the single human body image of the target human body is a front image, a deep learning network may be used to predict the human body back texture 41, and the human body back texture 41 and the human body front texture 42 in the single human body image may be used to perform texture mapping on the initial three-dimensional model, that is, texture reconstruction may be performed on the initial three-dimensional model. The three-dimensional model 43 in FIG. 4 has already been mapped with the above human body back and front textures on the initial three-dimensional model. The initial three-dimensional model obtained in step 104 is a mesh of a human body geometric structure. In this step, a human body texture is added to the model based on the mesh model. In addition, for some remaining invisible human body part regions, an interpolation technique may be used to fill some gaps in the model with texture to complement the texture of the initial three-dimensional model, thereby obtaining a three-dimensional human body model 44 of the target human body.

In the 3D human body reconstruction method of this embodiment, local geometric reconstruction is performed on a local part of a target human body, and a 3D mesh model of the local part obtained by the local geometric reconstruction is merged with a 3D mesh model of the target human body, so that the local part in the initial 3D model of the target human body becomes clearer, more delicate and more accurate, and the reconstruction effect of the local part is improved. In addition, in this method, reconstruction is performed based on a single human body image of the target human body, which simplifies the user's cooperation procedure and makes 3D human body reconstruction easier.

After a three-dimensional human body model of a human body is obtained, skinning weights for driving the three-dimensional human body model may be specified based on the three-dimensional human body model and the human body skeletal structure of a target human body. The skinning weights are used to drive the constructed three-dimensional human body model. For example, when trying to drive various operations of the three-dimensional human body model, it is necessary to bind the model to the human body skeletal structure. Binding the model to the skeleton in this way is skinning. The model can then be moved by the movement of the skeleton. The skinning weights are used to represent the magnitude of the influence of the joint points of the skeleton on the vertices of the model. Based on the skinning weights, it is possible to control the magnitude of the influence that each vertex in the three-dimensional human body model receives from each joint point of the skeleton, thereby better controlling the movement of the model.

Specifically, calculating the skinning weight of the three-dimensional human body model may include the following process. In step 100, a human body skeletal structure is obtained based on a single human body image of a target human body. In this step, the human body skeletal structure and the obtained three-dimensional human body model may be input into a deep learning network, and the skinning weight of the model may be automatically obtained through the deep learning network.

5, first, attribute features corresponding to each vertex in the 3D human body model 51 may be generated based on the 3D human body model 51 and the human body skeleton 52. The attribute features may be constructed using the spatial relationship between each vertex and the human body skeleton. For example, for one of the vertices, the attribute features of the vertex may include the following four features:
1) The position coordinates of the vertex.
2) The position coordinates of the K skeletal joint points closest to the vertex.
3) The geodesic distance from the position of the vertex to each of the K skeletal articulation points.
4) The angle between a vector pointing from each of the K skeletal joint points to the vertex and the skeleton on which the skeletal joint point is located, the vector starting from the starting point being one of the K skeletal joint points.
Here, K is a positive integer.

Still referring to FIG. 5, after the attribute features of each vertex are obtained, the attribute features of each vertex and the adjacent relationship features between each vertex may be input to a spatial graph convolutional attention network of a deep learning network. Before inputting these features to the spatial graph convolutional attention network, the features may be converted into hidden layer features by a multi-layer perceptron. The spatial graph convolutional attention network may predict the weight of influence of each vertex from each of the K skeletal articulation points based on the hidden layer features. A later multi-layer perceptron in the deep learning network may be used to perform a normalization process on the weight, so that for a vertex, the sum of the weights of influence of each skeletal articulation point on the vertex is 1. The weight of influence of each skeletal articulation point corresponding to each vertex in the finally obtained 3D human body model is the skinning weight of the vertex.

In the three-dimensional human body reconstruction method of this embodiment, a human body skeletal structure is obtained based on a single human body image of a target human body, and skinning weights can be automatically calculated based on the human body skeletal structure and the three-dimensional human body model obtained by reconstruction. Therefore, not only can the consistency of the semantic structure of the skeleton in different input images be guaranteed, but also appropriate skinning weights can be quickly generated taking into account different clothing and clothing shapes. Here, the semantic consistency of the skeleton can facilitate the registration of the model and the existing motion library, and the advantage of the semantic consistency is that it is easy to apply (register) the generated model and skeleton to the motion library. The motion library may pre-store several motion sequences of a person, such as dancing, boxing, etc. The motion library stores a series of moving skeletons. The meanings and structures of these skeletons in the motion library are consistent. If the generated skeleton has randomness (joint meaning is uncertain), applying the motion in the motion library is disadvantageous to the generated model. Therefore, in this embodiment, the consistency of the semantic structure of the generated skeleton is guaranteed, making it easier to register the motion library. The skinning weights calculated and generated according to the specific shapes can make the visual effect of the movement of different human body models more natural.

Another embodiment of the present invention provides a method for three-dimensional human body reconstruction. The difference between the reconstruction flow of this embodiment and the embodiment of FIG. 1 is that the flow of performing human body geometric reconstruction based on a single human body image of the target human body in step 100 is improved to improve the geometric reconstruction accuracy of the three-dimensional mesh model of the target human body obtained by reconstruction. In this embodiment, the same processing steps as in the embodiment of FIG. 1 will not be described in detail, and only the differences will be described.

As shown in FIG. 6, in addition to the network structure shown in FIG. 2, a second deep neural network branch 61 is added. The second deep neural network branch 61 may include a local feature sub-network 611 and a second fitting sub-network 612. A local image 62 may be obtained by extracting an image of a local region from a single human body image 21 of a target human body. The second deep neural network is for performing three-dimensional reconstruction on the local image 62.

It should be noted that the body region of the target body included in the local image here may not be exactly the same as the local portion corresponding to the local geometric reconstruction in step 102. For example, the local image here may include a region range from the shoulders of the target body or higher, while the local portion of the reconstruction in step 102 may be the face of the target body. Of course, performing reconstruction on the shoulders of the target body or higher in FIG. 6 is merely an example, and refined geometric reconstruction may be performed on other body regions of the target body.

Specifically, still referring to FIG. 6, a first human body model is obtained by performing reconstruction through the first deep neural network branch 22, a local image 62 is input to the second deep neural network branch 61, and feature extraction is performed on the local image through the local feature sub-network 611 to obtain a second image feature. Next, a second human body model is obtained through the second fitting sub-network 612 based on the second image feature and the intermediate feature output from the first fitting sub-network 222. The intermediate feature may be a feature output from a part of the network structure in the first fitting sub-network 222. For example, if the first fitting sub-network 222 includes a certain number of fully connected layers, the output of a part of the fully connected layers may be input to the second fitting sub-network 612 as the intermediate feature.

For example, the structure of the second deep neural network branch 61 may be basically the same as that of the first deep neural network branch 22. For example, the global feature sub-network 221 in the first deep neural network branch 22 may include four blocks, each of which may include a certain number of feature extraction layers, such as convolution layers, pooling layers, etc., while the local feature sub-network 611 in the second deep neural network branch 61 may include one of the above blocks. After the first human body model and the second human body model are obtained, the first human body model and the second human body model may then be fused to obtain a fused human body model. Then, a meshing process is performed on the fused human body model to obtain a three-dimensional mesh model of the target human body.

In the 3D human body reconstruction method of this embodiment, not only is local geometric reconstruction performed on a local portion of the target human body to improve the reconstruction effect of the local portion, but reconstruction is performed based on a single human body image of the target human body to simplify the user's collaboration procedure. In addition, the local image is reconstructed via a second deep neural network, thereby improving the reconstruction effect on the local human body region of the target human body.

Yet another embodiment of the present invention provides a method for 3D human body reconstruction. Compared with the embodiment of FIG. 1, the reconstruction flow of this yet another embodiment provides a method for predicting the texture of the rear surface of the human body through a specific deep learning network. In this embodiment, the same processing steps as in the embodiment of FIG. 1 will not be described in detail, and only the differences will be described.

As shown in FIG. 7, a single human body image of a target human body may include a background image and a front texture of the human body. In this case, image segmentation may be performed to extract the front texture of the human body, and then the back texture of the human body may be predicted based on the front texture. For example, a front image 71 of the target human body may be segmented to obtain a first segmented mask 72 and a front texture 73 of the target human body after segmentation. The first segmented mask 72 may be horizontally inverted to obtain a second segmented mask 74, and the front texture 73, the first segmented mask 72, and the second segmented mask 74 may be input to a texture generation network 75, and the back texture of the target human body output from the texture generation network 75 may finally be obtained.

In addition, while FIG. 7 illustrates an example in which the first segmented mask 72 is horizontally flipped to obtain the second segmented mask 74, this is not limited to the actual implementation. For example, a front image of the target human body may be input to a pre-trained neural network, which directly outputs the first and second segmented masks. After the front texture and back texture of the target human body are obtained, the front texture and back texture may be mapped onto an initial three-dimensional model of the human body to obtain a three-dimensional human body model of the target human body.

The training procedure of the texture generating network 75 may include the following processes. Referring to FIG. 8, a supporting texture generating network 76 may be used. The supporting texture generating network 76 may include a network structure of a part of the texture generating network 75. For example, the texture generating network 75 may be based on the supporting texture generating network 76 with a certain number of convolutional layers added.

During training, the assisting texture generation network may be trained based on the assisting human body image, the third sample segmentation mask, and the fourth sample segmentation mask in the training sample image set, and after the training of the assisting texture generation network is completed, at least some of the network parameters of the assisting texture generation network may be set as some of the initialization network parameters of the texture generation network, and the texture generation network may be trained based on the front texture of the human body sample, the first sample segmentation mask, and the second sample segmentation mask. Here, the assisting human body image is obtained by reducing the resolution of a single image of the human body sample. The first sample segmentation mask corresponds to a mask region of the front texture of the human body sample, the second sample segmentation mask corresponds to a mask region of the back texture of the human body sample, the third sample segmentation mask corresponds to a mask region of the front texture of the human body in the assisting human body image, and the fourth sample segmentation mask corresponds to a mask region of the back texture of the human body in the assisting human body image.

Continuing to refer to FIG. 8, image segmentation may be performed on the supporting human body image 81 to obtain a front texture 82 of the human body in the supporting human body image 81, a third sample segmentation mask 83, and a fourth sample segmentation mask 84, which may be input to the supporting texture generation network 76 to obtain a first predicted value of the back texture of the human body in the supporting human body image 81, and the network parameters of the supporting texture generation network 76 may be adjusted based on the first predicted value and the first true value of the back texture of the human body in the supporting human body image 81. By repeating this process multiple times, a training-completed supporting texture generation network 76 may be obtained. Here, the training supervision for the supporting texture generation network may include, in addition to the loss Loss calculated based on the first predicted value and the first true value, other losses based on the first predicted value, such as feature losses calculated based on the texture features of the supporting human body image and the first predicted value. The supporting human body image may be obtained by reducing the resolution of the human body front image 71 in FIG. 7. Therefore, the resolution of the front texture 82 of the human body in the supporting human body image 81 is also lower than the resolution of the front texture 73 in FIG. 7. The third sample division mask corresponds to a mask area of the front texture of the human body in the supporting human body image, and the fourth sample division mask corresponds to a mask area of the back texture of the human body in the supporting human body image.

After the training of the assisting texture generation network is completed, the network parameters of the assisting texture generation network may be used as the initialization of some of the network parameters of the texture generation network. That is, the network parameters of the texture generation network include at least some of the network parameters of the assisting texture generation network that has been trained. That is, the assisting texture generation network and the texture generation network share some of the network weights. Then, the human body front texture, the first sample division mask, and the second sample division mask in the training sample image set for training the texture generation network are input to the texture generation network to obtain a second predicted value of the back texture of the human body sample. The network parameters of the texture generation network are adjusted based on the second predicted value and the second true value of the back texture. The resolution of the second true value is higher than the resolution of the first true value, that is, the resolution of the back texture output from the texture generation network is slightly higher than the resolution of the back texture output from the assisting texture generation network.

In the 3D human body reconstruction method of this embodiment, not only is local geometric reconstruction performed on local parts of the target human body to improve the reconstruction effect of the local parts, but reconstruction is also performed based on a single human body image of the target human body to simplify the user's cooperation procedure. In addition, the texture prediction is automatically performed through a neural network to improve the generated texture effect. For example, the texture of the entire human body is made more uniform and the color is made more realistic. And by training the supporting texture generation network before training the texture generation network, the training procedure of the texture generation network is more stable and more likely to converge.

In another embodiment, in order to improve the effect of reconstruction, multiple images of the target human body from different angles may be acquired to comprehensively perform 3D reconstruction of the target human body. For example, take three images of the target human body as an example, and the three images may be collected at different angles. Referring to FIG. 2, the three images may be input to the global feature sub-network 221, and a first image feature corresponding to each of the three images may be obtained from the global feature sub-network 221. Then, the three first image features are fused, and the fused image feature is input to the first fitting sub-network 222 for further processing.

When the 3D human body reconstruction employs the network structure shown in FIG. 6, in addition to using each of the above three images as an input to the global feature sub-network 221, local regions may be extracted from the three images to obtain local images, each of the three local images may be used as an input to the local feature sub-network 611, second image features corresponding to each of the three local images output from the local feature sub-network 611 may be obtained, and the three second image features may then be fused, and the image features obtained by the fusion may be further processed as an input to the second fitting sub-network 612.

As described above, by acquiring multiple images of the target human body from different angles and comprehensively reconstructing the target human body in three dimensions, it is possible to obtain a more detailed three-dimensional human body model corresponding to the target human body.

It should also be noted that in each flow step of the 3D human body reconstruction method described in any embodiment of the present invention, any of the neural network models may be trained separately. For example, the first deep neural network branch and the texture generation network may each be trained separately.

Below, an example of one 3D human body reconstruction flow is described. Note that the same processing steps as those described in any of the above method embodiments are briefly described here, and for detailed procedures, please refer to the above embodiments.

In this example, assuming that a three-dimensional human body model of user U1 is to be constructed based on a single human body image of user U1, the single human body image may be a frontal image of user U1, and includes a frontal texture and a background image of user U1. Referring to FIG. 9, the single human body image 91 of user U1 includes a frontal texture 92 and a background image 93 of the user.

First, two types of reconstruction may be performed based on a single human body image 91 of user U1.

One aspect of the reconstruction is to perform human body geometric reconstruction based on a single human body image 91 to obtain a three-dimensional mesh model and a human body skeletal structure of U1. For example, the single human body image 91 may be processed through the network shown in FIG. 6, the single human body image 91 may be processed through a global feature sub-network and a first fitting sub-network in a first deep neural network branch to obtain a first human body model, and an image of a region above the human shoulders in the single human body image 91 may be processed through a local feature sub-network and a second fitting sub-network in a second deep neural network branch to obtain a second human body model. After fusing the first human body model and the second human body model, a fused human body model is obtained. A meshing process is performed on the fused human body model to obtain a three-dimensional mesh model (mesh) of the user U1.

Another aspect of reconstruction is to perform local geometric reconstruction on the face of user U1 based on a single human body image 91 to obtain a 3D mesh model of the face. Specifically, feature extraction may be performed on the single human body image 91, and the extracted image features and a 3D facial topology template may be input to a graph convolutional neural network to obtain the face mesh of user U1.

Next, the face mesh (three-dimensional mesh model of the face) obtained by the above reconstruction may be combined with the body mesh of user U1 (three-dimensional mesh model of U1's body) and fused to obtain an initial three-dimensional model of U1.

Specifically, based on the schematic flow of FIG. 3, the key points of the face may be considered, the identifiers and positions of the corresponding model key points in the face mesh and the human body mesh may be identified, and a coordinate transformation relationship between the models may be identified based on the identifiers and positions of these model key points and parameters such as the external camera parameters of the models. Based on the coordinate transformation relationship, the face mesh is transformed into the coordinate system of the human body mesh, the face in the human body mesh is replaced with the face mesh, and the face mesh and the human body mesh are fused by Poisson reconstruction to obtain an initial three-dimensional model of user U1.

Then, the human body texture of U1 is reconstructed based on the initial 3D model and the single human body image 91 of user U1. Here, since the single human body image 91 is the front texture of user U1, the back texture of U1 may be predicted based on the front texture.

Specifically, a single human body image 91 may be subjected to human body segmentation to obtain a human body front texture from which the background image has been removed and a first segmentation mask for representing the human body front texture region, and the first segmentation mask may be inverted to obtain a second segmentation mask for representing the human body rear texture region. Next, the human body front texture, the first segmentation mask, and the second segmentation mask may be input into a pre-trained texture generation network to obtain the rear texture of user U1. Finally, texture mapping may be performed on the initial three-dimensional model based on the front texture and the rear texture, and the gap regions of the model may be filled and complemented with texture, to finally obtain a three-dimensional human body model of U1 having texture.

To facilitate model driving of the constructed 3D human body model, the 3D human body model of U1 obtained by reconstruction and the human body skeletal structure obtained when reconstructing the 3D mesh model of U1 may be used to calculate skinning weights for the 3D human body model. The model may then be driven to perform an action using the skinning weights.

Figure 10 illustrates a structural schematic diagram of a 3D human body reconstruction device. As shown in Figure 10, the device may include a global reconstruction module 1001, a local reconstruction module 1002, a fusion processing module 1003, and a texture reconstruction module 1004.

The overall reconstruction module 1001 performs body geometry reconstruction based on a single body image of a target body to obtain a three-dimensional mesh model of the target body.

The local reconstruction module 1002 performs local geometric reconstruction of a local portion of the target body based on a single body image of the target body, and obtains a three-dimensional mesh model of the local portion.

The fusion processing module 1003 fuses the 3D mesh model of the local area with the 3D mesh model of the target human body to obtain an initial 3D model.

The texture reconstruction module 1004 reconstructs the body texture of the target body based on the initial 3D model and the single body image to obtain a 3D body model of the target body.

In one example, when obtaining a three-dimensional mesh model of the target human body, the overall reconstruction module 1001 is specifically used to perform three-dimensional reconstruction on a single human body image of the target human body via a first deep neural network branch to obtain a first human body model, perform three-dimensional reconstruction on a local image in the single human body image via a second deep neural network branch to obtain a second human body model, fuse the first human body model and the second human body model to obtain a fused human body model, and perform a meshing process on the fused human body model to obtain a three-dimensional mesh model of the target human body. The local image includes a local region of the target human body.

In one example, the local reconstruction module 1002 is specifically used to perform feature extraction on a single human body image of the target human body, obtain third image features, and identify a three-dimensional mesh model of the local region based on the third image features and a three-dimensional topology template of the local region.

In one example, the fusion processing module 1003 is specifically used to obtain multiple key points of the local region based on a single human body image of the target human body, identify information of first model key points corresponding to the multiple key points in a three-dimensional mesh model of the target human body, and identify information of second model key points corresponding to the multiple key points in the three-dimensional mesh model of the local region, and fuse the three-dimensional mesh model of the local region with the three-dimensional mesh model of the target human body based on the information of the first model key points and the information of the second model key points to obtain the initial three-dimensional model.

In one example, the fusion processing module 1003 is used to fuse the 3D mesh model of the local region with the 3D mesh model of the target human body based on the information of the first model keypoints and the information of the second model keypoints to obtain the initial 3D model. Specifically, the fusion processing module 1003 is used to determine a coordinate transformation relationship between the 3D mesh model of the target human body and the 3D mesh model of the local region based on the information of the first model keypoints and the information of the second model keypoints, transform the 3D mesh model of the local region into the coordinate system of the 3D mesh model of the target human body based on the coordinate transformation relationship, fuse the 3D mesh model of the local region with the 3D mesh model of the target human body in the transformed coordinate system, and obtain the initial 3D model.

In one example, the texture reconstruction module 1004 specifically performs body segmentation on the single human body image, obtains a first segmentation mask, a second segmentation mask, and a front texture of a target human body, inputs the front texture, the first segmentation mask, and the second segmentation mask into a texture generation network to obtain a back texture of the target human body, and is used to obtain a three-dimensional human body model having texture corresponding to the target human body based on the back texture and the front texture, where the first segmentation mask corresponds to a mask area of the front texture and the second segmentation mask corresponds to a mask area of the back texture of the target human body.

In one example, as shown in FIG. 11, the device may further include a model training module 1005.

The model training module 1005 is for training the texture generation network, and is specifically used to perform body segmentation on a single image of a human body sample in a training sample image set, obtain a first sample segmentation mask, a second sample segmentation mask, and a frontal texture of the human body sample, train the supporting texture generation network based on the frontal texture of the human body in the supporting human body image obtained by reducing the resolution of the single image of the human body sample, a third sample segmentation mask, and a fourth sample segmentation mask, and after the training of the supporting texture generation network is completed, train the texture generation network based on the frontal texture of the human body sample, the first sample segmentation mask, and the second sample segmentation mask. The first sample division mask corresponds to a mask region of the front texture of the human body sample, the second sample division mask corresponds to a mask region of the back texture of the human body sample, the third sample division mask corresponds to a mask region of the front texture of the human body in the supporting human body image, and the fourth sample division mask corresponds to a mask region of the back texture of the human body in the supporting human body image, and the network parameters of the texture generation network include at least a portion of the network parameters of the supporting texture generation network that has completed training.

In some embodiments, the apparatus may perform any of the methods described above, which will not be repeated here for the sake of brevity.

An embodiment of the present invention further provides an electronic device. The electronic device includes a memory and a processor, the memory being adapted to store computer-readable instructions, and the processor being adapted to implement a method of any embodiment of the present specification by invoking the computer instructions.

An embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a method according to any of the embodiments of the present specification is performed.

As will be appreciated by those skilled in the art, one or more embodiments of the present invention may be provided as a method, a system, or a computer program product. The computer program product includes a computer program that, when executed by a processor, may perform the method of any of the embodiments herein. Thus, one or more embodiments of the present invention may take the form of a 100% hardware embodiment, a 100% software embodiment, or an embodiment that combines software and hardware aspects. Also, one or more embodiments of the present invention may take the form of a computer program product embodied on one or more computer usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) that includes computer usable program code.

The term "and/or" in the examples of the present invention means that at least one of the two is present. For example, "A and/or B" includes the three forms A, B, and "A and B."

Each embodiment of the present invention is described in an incremental manner, with the focus being on the differences between each embodiment and the other embodiments, and reference may be made to the same or similar parts between the embodiments. In particular, the data processing device embodiment is essentially similar to the method embodiment, and therefore the description is relatively simple, and reference may be made to the partial description of the method embodiment for the relevant parts.

Specific embodiments of the present invention have been described above. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in an order different from that shown in the examples and still achieve desirable results. Also, the procedures depicted in the figures do not necessarily require the particular order or sequential order shown to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or advantageous.

The subject matter and functional operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and structural equivalents thereof, or any combination of one or more of them. The subject matter described herein may be implemented as one or more modules in one or more computer programs, i.e., computer program instructions coded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions may be coded into an artificially generated transmission signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or any combination of one or more of them.

The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs that operate on input data to generate output and perform corresponding functions. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and devices may also be implemented as special purpose logic circuitry.

A computer suitable for executing a computer program may include, for example, a general-purpose and/or dedicated microprocessor, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory and/or a random access memory. The basic components of a computer include a central processing unit for implementing and executing instructions, and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or the computer is operatively coupled to the mass storage device to receive and transmit data, or the two situations may be combined. However, a computer does not necessarily have such devices. A computer may also be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash memory driver. These are just a few examples.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or integrated into special purpose logic circuitry.

Although the present invention includes a large number of specific implementation details, these details should not be construed as limiting any of the scope of the disclosure or the scope of the claimed protection, but are used primarily to describe the features of certain disclosed specific embodiments. Some features described in multiple embodiments of the present invention may be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may be implemented separately in multiple embodiments or in any suitable subcombination. Also, although features may play a role in several combinations as described above and are initially claimed as such, one or more features from the claimed combination may in some cases be removed from the combination, and the claimed combination may refer to a subcombination or a variation of the subcombination.

Similarly, although operations are depicted in a particular order in the figures, this should not be understood as requiring that these operations be performed in the particular order or sequence shown, or that all of the illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. Also, the separation of various system modules and units in the above examples should not be understood as requiring such separation in all embodiments. Moreover, as will be appreciated, the program units and systems described may typically be integrated into a single software product or packaged as multiple software products.

Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Also, the processes depicted in the figures do not necessarily achieve desirable results in the particular order or sequential order shown. In some embodiments, multitasking and parallel processing may be advantageous.

The above are merely preferred embodiments of one or more embodiments of the present invention, and are not intended to limit the one or more embodiments of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of the present invention should be included within the protection scope of one or more embodiments of the present invention.

21 Human body image 22 First deep neural network branch 31 Key point 41 Human body back texture 42 Human body front texture 43 Three-dimensional model 44 Three-dimensional human body model 51 Three-dimensional human body model 52 Human body skeletal structure 61 Second deep neural network branch 62 Local image 71 Front image 72 First segmentation mask 73 Front texture 74 Second segmentation mask 75 Texture generation network 81 Support human body image 82 Front texture 83 Third sample segmentation mask 84 Fourth sample segmentation mask 91 Human body image 92 Front texture 93 Background image 221 Global feature sub-network 222 First fitting sub-network 611 Local feature sub-network 612 Second fitting sub-network 1001 Global reconstruction module 1002 Local reconstruction module 1003 Fusion processing module 1004 Texture reconstruction module 1005 Model training module

A fifth aspect provides a computer program comprising computer readable instructions which, when executed by a processor, perform a method according to any embodiment of the present invention.

Claims (20)

performing human body geometry reconstruction based on the human body image of the target human body to obtain a 3D mesh model of the target human body;
performing local geometric reconstruction on a local portion of the target human body based on a human body image of the target human body to obtain a three-dimensional mesh model of the local portion;
Fusing the 3D mesh model of the local region with the 3D mesh model of the target human body to obtain an initial 3D model;
and reconstructing the body texture of the target body based on the initial 3D model and the human body image to obtain a 3D human body model of the target body. The step of performing human body geometry reconstruction based on the human body image of the target human body to obtain a three-dimensional mesh model of the target human body includes:
performing 3D reconstruction on the human body image of the target human body through a first deep neural network branch to obtain a first human body model;
performing 3D reconstruction on a local image in the human body image through a second deep neural network branch to obtain a second human body model;
fusing the first human body model and the second human body model to obtain a fused human body model;
performing a meshing process on the fused human body model to obtain a three-dimensional mesh model of the target human body;
The method of claim 1 , wherein the local images include local regions of the target body. the first deep neural network branch includes a global feature sub-network and a first fitting sub-network, and the second deep neural network branch includes a local feature sub-network and a second fitting sub-network;
The step of performing 3D reconstruction on the human body image of the target human body via the first deep neural network branch to obtain a first human body model includes: performing feature extraction on the human body image via the global feature sub-network to obtain first image features; and obtaining the first human body model based on the first image features via the first fitting sub-network;
3. The method of claim 2, wherein the step of performing 3D reconstruction on a local image in the human body image via the second deep neural network branch to obtain a second human body model includes: performing feature extraction on the local image via the local feature sub-network to obtain second image features; and obtaining the second human body model via the second fitting sub-network based on the second image features and intermediate features output from the first fitting sub-network. The step of performing local geometric reconstruction on a local portion of the target human body based on a human body image of the target human body to obtain a three-dimensional mesh model of the local portion includes:
performing feature extraction on the human body image of the target human body to obtain a third image feature;
The method of any one of claims 1 to 3, further comprising: identifying a three-dimensional mesh model of the local area based on the third image feature and a three-dimensional topological template of the local area. The step of fusing the 3D mesh model of the local region and the 3D mesh model of the target human body to obtain an initial 3D model includes:
obtaining a plurality of key points of the local region based on a human body image of the target human body;
Identifying information of first model key points corresponding to the plurality of key points in a three-dimensional mesh model of the target human body, and identifying information of second model key points corresponding to the plurality of key points in a three-dimensional mesh model of the local region;
The method according to any one of claims 1 to 4, further comprising: fusing the 3D mesh model of the local area with a 3D mesh model of the target body based on information of the first model keypoints and information of the second model keypoints to obtain the initial 3D model. Fusing the 3D mesh model of the local region with the 3D mesh model of the target human body based on the information of the first model keypoints and the information of the second model keypoints to obtain the initial 3D model,
determining a coordinate transformation relationship between the 3D mesh model of the target human body and the 3D mesh model of the local region based on information of the first model key points and information of the second model key points;
transforming the 3D mesh model of the local region into a coordinate system of the 3D mesh model of the target human body based on the coordinate transformation relationship;
and fusing the 3D mesh model of the local area with a 3D mesh model of the target body in the transformed coordinate system to obtain the initial 3D model. The human body image includes a front texture of the target human body and a background image;
The step of reconstructing a body texture of the target human body based on the initial 3D model and the human body image to obtain a 3D human body model of the target human body includes:
performing a human body segmentation on the human body image to obtain a first segmentation mask, a second segmentation mask, and a front texture of the target human body;
inputting the front texture, the first segmentation mask, and the second segmentation mask into a texture generation network to obtain a rear texture of the target human body;
and obtaining a textured three-dimensional human body model corresponding to the target human body based on the back texture and the front texture;
7. The method of claim 1, wherein the first segmentation mask corresponds to a mask area of the front texture and the second segmentation mask corresponds to a mask area of the back texture of the target body. Training the texture generation network includes:
performing body segmentation on an image of a body sample in a training sample image set to obtain a first sample segmentation mask, a second sample segmentation mask, and a front texture of the body sample;
A process of training an assisting texture generation network based on a front texture of the human body in the assisting human body image obtained by reducing the resolution of the image of the human body sample, a third sample division mask, and a fourth sample division mask;
After the training of the auxiliary texture generation network is completed, a process of training the texture generation network based on the front texture of the human body sample, the first sample segmentation mask, and the second sample segmentation mask;
The first sample division mask corresponds to a mask area of a front texture of the human body sample, and the second sample division mask corresponds to a mask area of a back texture of the human body sample;
The third sample division mask corresponds to a mask area of a front texture of a human body in the supporting human body image, and the fourth sample division mask corresponds to a mask area of a back texture of a human body in the supporting human body image;
The method of claim 7 , wherein the network parameters of the texture generating network include network parameters of at least a portion of the supporting texture generating network that has completed training. The local area of the target body is a face of the target body, and/or
The method according to any one of claims 1 to 8, wherein the human body image is an RGB image. obtaining a human body skeleton structure of the target human body when performing human body geometry reconstruction based on a human body image of the target human body;
The method according to any one of claims 1 to 9, further comprising: after a three-dimensional human body model of the target human body is obtained, determining a skinning weight for driving the three-dimensional human body model based on the three-dimensional human body model and the human body skeletal structure. a global reconstruction module for performing human body geometric reconstruction based on the human body image of a target human body to obtain a three-dimensional mesh model of the target human body;
a local reconstruction module for performing local geometric reconstruction on a local portion of the target human body based on a human body image of the target human body to obtain a three-dimensional mesh model of the local portion;
a fusion processing module for fusing the 3D mesh model of the local region and the 3D mesh model of the target human body to obtain an initial 3D model;
a texture reconstruction module for reconstructing the body texture of the target body based on the initial 3D model and the human body image, and obtaining a 3D human body model of the target body. When obtaining the 3D mesh model of the target human body, the overall reconstruction module specifically includes:
Perform 3D reconstruction on the human body image of the target human body through a first deep neural network branch to obtain a first human body model;
Performing 3D reconstruction on a local image in the human body image via a second deep neural network branch to obtain a second human body model;
Fusing the first human body model and the second human body model to obtain a fused human body model;
A meshing process is performed on the fused human body model to obtain a three-dimensional mesh model of the target human body;
The apparatus of claim 11 , wherein the local image includes a local region of the target body. Specifically, the local reconstruction module:
Perform feature extraction on the human body image of the target human body to obtain a third image feature;
13. The apparatus according to claim 11 or 12, adapted to identify a 3D mesh model of the local area based on the third image feature and a 3D topological template of the local area. The fusion processing module specifically includes:
Obtaining a plurality of key points of the local region based on a human body image of the target human body;
Identifying information of first model key points corresponding to the plurality of key points in a three-dimensional mesh model of the target human body, and identifying information of second model key points corresponding to the plurality of key points in a three-dimensional mesh model of the local region;
The apparatus according to any one of claims 11 to 13, characterized in that it is used to fuse a 3D mesh model of the local area with a 3D mesh model of the target body based on information of the first model keypoints and information of the second model keypoints to obtain the initial 3D model. The fusion processing module is configured to fuse the 3D mesh model of the local region with the 3D mesh model of the target body based on the information of the first model keypoints and the information of the second model keypoints to obtain the initial 3D model, specifically:
Identifying a coordinate transformation relationship between the 3D mesh model of the target human body and the 3D mesh model of the local region based on the information of the first model keypoints and the information of the second model keypoints;
Transforming the 3D mesh model of the local region into a coordinate system of the 3D mesh model of the target body based on the coordinate transformation relationship;
The apparatus according to claim 14, further comprising: a 3D mesh model of the local area in a transformed coordinate system, the 3D mesh model being adapted to fuse with a 3D mesh model of the target body to obtain the initial 3D model. Specifically, the texture reconstruction module:
Performing a human body segmentation on the human body image to obtain a first segmentation mask, a second segmentation mask, and a front texture of the target human body;
Input the front texture, the first segmentation mask, and the second segmentation mask into a texture generation network to obtain a rear texture of the target human body;
to obtain a textured three-dimensional human body model corresponding to the target human body based on the back texture and the front texture;
16. The apparatus of claim 11, wherein the first segmentation mask corresponds to a mask area of the front texture and the second segmentation mask corresponds to a mask area of a back texture of the target body. The 3D human body reconstruction apparatus further comprises a model training module for training the texture generation network;
The model training module specifically includes:
Performing body segmentation on an image of a human body sample in the training sample image set to obtain a first sample segmentation mask, a second sample segmentation mask, and a front texture of the human body sample;
Training an assisting texture generation network based on the front texture of the human body in the assisting human body image obtained by reducing the resolution of the image of the human body sample, a third sample segmentation mask, and a fourth sample segmentation mask;
After the training of the auxiliary texture generation network is completed, the texture generation network is trained based on the front texture of the human body sample, the first sample segmentation mask, and the second sample segmentation mask;
The apparatus of claim 16, wherein the first sample division mask corresponds to a mask region of a front texture of the human body sample, the second sample division mask corresponds to a mask region of a back texture of the human body sample, the third sample division mask corresponds to a mask region of a front texture of the human body in the supporting human body image, and the fourth sample division mask corresponds to a mask region of a back texture of the human body in the supporting human body image, and the network parameters of the texture generation network include at least a portion of the network parameters of the supporting texture generation network that has completed training. 1. An electronic device comprising:
A memory and a processor,
11. An electronic device, characterized in that the memory is adapted to store computer readable instructions and the processor is adapted to implement the method according to any one of claims 1 to 10 by invoking the computer readable instructions. A computer-readable storage medium storing a computer program,
A computer-readable storage medium, comprising a computer program, the computer program being executed by a processor to perform the method according to any one of claims 1 to 10. A computer program product comprising a computer program,
A computer program product, characterized in that the method according to any one of claims 1 to 10 is performed when the computer program is executed by a processor.