CN111783986A - Network training method and device, attitude prediction method and device - Google Patents

Abstract

The present disclosure relates to a network training method and device, and an attitude prediction method and device. The method includes: predicting a two-dimensional sample image through an attitude prediction network, and obtaining a predicted segmentation mask corresponding to a target object in the two-dimensional sample image and the predicted posture information corresponding to the target object, the predicted posture information includes three-dimensional rotation information and three-dimensional translation information; according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, a differentiable rendering operation is performed, Obtaining differentiable rendering information corresponding to the target object; determining the posture prediction according to the two-dimensional sample image, the predicted segmentation mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information The total loss of the self-supervised training of the network; the attitude prediction network is trained according to the total loss of the self-supervised training, and the embodiment of the present disclosure can improve the accuracy of the attitude prediction network.

Description

Network training method and device, attitude prediction method and device

technical field

The present disclosure relates to the field of computer technology, and in particular, to a network training method and device, and a posture prediction method and device.

Background technique

Obtaining the six-dimensional (6D) pose of an object in three-dimensional (3D) space from a two-dimensional (2D) image (ie, 3-DOF rotation and 3-DOF translation) is critical in many real-world applications, such as for Tasks such as grasping or motion planning of robots provide critical information; in unmanned driving, obtaining 6D poses of vehicles and pedestrians can provide driving decision-making information for vehicles.

In recent years, deep learning has made great progress in the task of 6D pose estimation. However, it is still very challenging to estimate the 6D pose of an object using only monocular RGB (red\green\blue, red\green\blue) images. sexual task. One of the important reasons is that the amount of data required for deep learning is very large, and the acquisition of real labeled data for 6D object pose estimation is very complicated, which is very time-consuming and labor-intensive.

SUMMARY OF THE INVENTION

The present disclosure proposes a self-supervised training technical solution for training a neural network.

According to an aspect of the present disclosure, a network training method is provided, comprising:

The two-dimensional sample image is predicted by the attitude prediction network, and the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted posture information corresponding to the target object are obtained. The predicted posture information includes three-dimensional rotation information and three-dimensional translation information;

Perform a differentiable rendering operation according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, to obtain differentiable rendering information corresponding to the target object;

determining the total loss of self-supervised training of the pose prediction network according to the two-dimensional sample image, the predicted segmentation mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information;

The pose prediction network is trained according to the self-supervised training total loss.

In a possible implementation manner, the differentiable rendering information corresponding to the target object includes: rendering a segmentation mask, rendering a two-dimensional image, and rendering a depth image,

Determining the total loss of self-supervised training of the attitude prediction network according to the two-dimensional sample image, the predicted segmentation mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information, including:

determining a first self-supervised training loss according to the two-dimensional sample image and the rendered two-dimensional image;

determining a second self-supervised training loss according to the predicted segmentation mask and the rendered segmentation mask;

determining a third self-supervised training loss according to the depth image corresponding to the two-dimensional sample image and the rendered depth image;

According to the first self-supervised training loss, the second self-supervised training loss and the third self-supervised training loss, a total self-supervised training loss of the pose prediction network is determined.

In a possible implementation manner, the determining the first self-supervised training loss according to the two-dimensional sample image and the rendered two-dimensional image includes:

After the two-dimensional sample image and the rendered two-dimensional image are respectively converted into the color model LAB mode, according to the two-dimensional sample image after the conversion mode, the rendered two-dimensional image after the conversion mode, and the predicted segmentation mask, using the first loss function determines the first image loss;

According to the two-dimensional sample image, the rendered two-dimensional image and the predicted segmentation mask, a second loss function is used to determine a second image loss, and the second loss function is a loss function based on a multi-scale structural similarity index ;

According to the 2D sample image, the rendered 2D image and the predicted segmentation mask, a third loss function is used to determine a third image loss, and the third loss function is a multi-scale feature based on a deep convolutional neural network The loss function of distance;

The first self-supervised training loss is determined from the first image loss, the second image loss, and the third image loss.

In a possible implementation manner, the determining of the second self-supervised training loss according to the predicted segmentation mask and the rendering segmentation mask includes:

Based on the predicted segmentation mask and the rendered segmentation mask, a cross-entropy loss function is used to determine a second self-supervised training loss.

In a possible implementation manner, determining the third self-supervised training loss according to the depth image corresponding to the two-dimensional sample image and the rendered depth image includes:

Perform a back-projection operation on the depth image corresponding to the two-dimensional sample image and the rendered depth image, respectively, to obtain point cloud information corresponding to the depth image and point cloud information corresponding to the rendered depth image;

A third self-supervised training loss is determined according to the point cloud information corresponding to the depth image and the point cloud information corresponding to the rendered depth image.

In a possible implementation manner, the posture prediction network includes: a category prediction sub-network, a bounding box prediction sub-network, and a posture prediction sub-network,

The two-dimensional sample image is predicted through the attitude prediction network, and the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted attitude information corresponding to the target object are obtained, including:

Predict the two-dimensional sample image through the category prediction sub-network, and obtain category information corresponding to the target object in the two-dimensional sample image;

Predict the two-dimensional sample image through the bounding box prediction sub-network, and obtain bounding box information corresponding to the target object in the two-dimensional sample image;

The 2D sample image, the category information and the bounding box information are processed by the pose prediction sub-network to obtain the predicted segmentation mask corresponding to the target object in the 2D sample image and the corresponding target object the predicted pose information.

In a possible implementation manner, before the two-dimensional sample image is predicted by the pose prediction network, the method further includes:

Rendering and synthesizing operations are performed according to the 3D model of the object and the preset posture information to obtain a synthesized 2D image and annotation information of the synthesized 2D image, where the annotation information of the synthesized 2D image includes annotated object category information and annotated bounding box information , preset pose information and preset synthetic segmentation mask;

Predict the synthetic 2D image through the pose prediction network to obtain prediction information of the synthetic 2D image, where the prediction information includes predicted object category information, predicted bounding box information, predicted synthetic segmentation mask, and predicted synthetic attitude information;

The pose prediction network is trained according to the prediction information and the annotation information of the synthesized two-dimensional image.

According to an aspect of the present disclosure, there is provided a gesture prediction method, the method comprising:

The image to be processed is predicted and processed by the attitude prediction network, and the attitude information of the target object in the image to be processed is obtained,

Wherein, the posture prediction network is obtained by using the network training method described in any one of claims 1 to 7.

According to an aspect of the present disclosure, a network training apparatus is provided, comprising:

The prediction module is used to predict the two-dimensional sample image through the attitude prediction network, and obtain the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted attitude information corresponding to the target object, and the predicted attitude information includes 3D rotation information and 3D translation information;

a rendering module, configured to perform a differentiable rendering operation according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, and obtain differentiable rendering information corresponding to the target object;

A determination module, configured to determine the total loss of self-supervised training of the attitude prediction network according to the two-dimensional sample image, the predicted segmentation mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information ;

A self-supervised training module for training the pose prediction network according to the self-supervised training total loss.

In a possible implementation manner, the differentiable rendering information corresponding to the target object includes: rendering a segmentation mask, rendering a two-dimensional image, and rendering a depth image, and the determining module is further configured to:

determining a first self-supervised training loss according to the two-dimensional sample image and the rendered two-dimensional image;

determining a second self-supervised training loss according to the predicted segmentation mask and the rendered segmentation mask;

determining a third self-supervised training loss according to the depth image corresponding to the two-dimensional sample image and the rendered depth image;

According to the first self-supervised training loss, the second self-supervised training loss and the third self-supervised training loss, a total self-supervised training loss of the pose prediction network is determined.

In a possible implementation manner, the determining module is further configured to:

After the two-dimensional sample image and the rendered two-dimensional image are respectively converted into the color model LAB mode, according to the two-dimensional sample image after the conversion mode, the rendered two-dimensional image after the conversion mode, and the predicted segmentation mask, using the first loss function determines the first image loss;

According to the two-dimensional sample image, the rendered two-dimensional image and the predicted segmentation mask, a second loss function is used to determine a second image loss, and the second loss function is a loss function based on a multi-scale structural similarity index ;

According to the 2D sample image, the rendered 2D image and the predicted segmentation mask, a third loss function is used to determine a third image loss, and the third loss function is a multi-scale feature based on a deep convolutional neural network The loss function of distance;

The first self-supervised training loss is determined from the first image loss, the second image loss, and the third image loss.

In a possible implementation manner, the determining module is further configured to:

Based on the predicted segmentation mask and the rendered segmentation mask, a cross-entropy loss function is used to determine a second self-supervised training loss.

In a possible implementation manner, the determining module is further configured to:

Perform a back-projection operation on the depth image corresponding to the two-dimensional sample image and the rendered depth image, respectively, to obtain point cloud information corresponding to the depth image and point cloud information corresponding to the rendered depth image;

A third self-supervised training loss is determined according to the point cloud information corresponding to the depth image and the point cloud information corresponding to the rendered depth image.

In a possible implementation manner, the posture prediction network includes: a category prediction sub-network, a bounding box prediction sub-network, and a posture prediction sub-network, and the prediction module is further configured to:

Predict the two-dimensional sample image through the category prediction sub-network, and obtain category information corresponding to the target object in the two-dimensional sample image;

Predict the two-dimensional sample image through the bounding box prediction sub-network, and obtain bounding box information corresponding to the target object in the two-dimensional sample image;

The 2D sample image, the category information and the bounding box information are processed by the pose prediction sub-network to obtain the predicted segmentation mask corresponding to the target object in the 2D sample image and the corresponding target object the predicted pose information.

In a possible implementation, the apparatus further includes:

The pre-training module is used to perform a rendering and synthesis operation according to the three-dimensional model of the object and the preset posture information, and obtain a synthesized two-dimensional image and label information of the synthesized two-dimensional image, where the label information of the synthesized two-dimensional image includes the label object category information, annotated bounding box information, preset pose information, and preset synthetic segmentation masks;

Predict the synthetic 2D image through the pose prediction network to obtain prediction information of the synthetic 2D image, where the prediction information includes predicted object category information, predicted bounding box information, predicted synthetic segmentation mask, and predicted synthetic attitude information;

The pose prediction network is trained according to the prediction information and the annotation information of the synthesized two-dimensional image.

According to an aspect of the present disclosure, there is provided an apparatus for attitude prediction, the apparatus comprising:

The prediction module is used to perform prediction processing on the image to be processed through the attitude prediction network, and obtain the attitude information of the target object in the image to be processed,

Wherein, the posture prediction network is obtained by training using any one of the network training methods described above.

According to an aspect of the present disclosure, there is provided an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the above method when executed by a processor.

In this way, the two-dimensional sample image can be predicted through a posture prediction network, and the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted posture information corresponding to the target object can be obtained, and the predicted posture information includes three-dimensional rotation. information and three-dimensional translation information, and perform a differentiable rendering operation according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, and obtain the differentiable rendering information corresponding to the target object. According to the two-dimensional sample image, the predicted segmentation mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information, determine the total loss of training self-supervised training of the pose prediction network, and according to the The pose prediction network is trained with the self-supervised training total loss. According to the network training method and device, and the attitude prediction method and device provided by the embodiments of the present disclosure, the attitude prediction network can be improved in the accuracy of the attitude prediction network by self-supervised training of the attitude prediction network on the two-dimensional sample images and depth images without label information. At the same time, the training efficiency of the pose prediction network is improved.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the technical solutions of the present disclosure.

1 shows a flowchart of a network training method according to an embodiment of the present disclosure;

2 shows a schematic diagram of a network training method according to an embodiment of the present disclosure;

3 shows a schematic diagram of a network training method according to an embodiment of the present disclosure;

4 shows a schematic diagram of a network training method according to an embodiment of the present disclosure;

5 shows a block diagram of a network training apparatus according to an embodiment of the present disclosure;

FIG. 6 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure;

FIG. 7 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

In addition, in order to better illustrate the present disclosure, numerous specific details are set forth in the following detailed description. It will be understood by those skilled in the art that the present disclosure may be practiced without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present disclosure.

In the related art, when training a neural network for object pose estimation, a three-dimensional model of a known object is usually used to obtain a large amount of synthetic data by rendering, and then the neural network is trained through the synthetic data. However, there is a large domain gap between synthetic data and real data. Therefore, the results of training only on synthetic data are often not accurate and unsatisfactory. The effects of domain adaptation or domain randomization on this problem are also not satisfactory. very limited.

The embodiments of the present disclosure provide a method for self-supervised training of a network, which can improve the prediction accuracy of the attitude prediction network by self-supervised training of an attitude prediction network through real two-dimensional sample images and depth images.

1 shows a flowchart of a network training method according to an embodiment of the present disclosure. In a possible implementation, the network training method may be executed by an electronic device such as a terminal device or a server, and the terminal device may be a user equipment (User Equipment). Equipment, UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, Personal Digital Assistant (PDA), handheld devices, computing devices, in-vehicle devices, wearable devices, etc., the method can be processed by It is implemented in a manner that the processor invokes computer readable instructions stored in the memory. Alternatively, the method may be performed by a server.

As shown in Figure 1, the network training method includes:

In step S11, the two-dimensional sample image is predicted through a posture prediction network, and the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted posture information corresponding to the target object are obtained, and the predicted posture information includes 3D rotation information and 3D translation information.

For example, the pose prediction network is a neural network that predicts the 6D pose of a target object, which can be applied to fields such as "robot work", "autonomous driving", and "augmented reality". The above-mentioned two-dimensional sample image may be an image including a target object, and the target object may be any object, such as a human face, a human body, an animal, a plant, an object, and other objects.

The two-dimensional sample image can be input into the posture prediction network for prediction, and the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted posture information corresponding to the target object can be obtained, wherein the predicted segmentation mask of any pixel point. The pixel value is used to identify whether the pixel is a pixel in the target object in the sample two-dimensional image. For example, when the pixel value is 1, it identifies the pixel as a pixel on the target object, and when the pixel value is 0, it identifies the pixel. Pixels are not pixels on the target object. The predicted pose information may include three-dimensional rotation information R of the target object in three dimensions, which may be represented by quaternions, and three-dimensional translation information t (t _x , _ty , _tz ) of the target object in three dimensions.

In step S12, a differentiable rendering operation is performed according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, and differentiable rendering information corresponding to the target object is obtained.

For example, the two-dimensional sample image can be detected, the three-dimensional model corresponding to the target object in the two-dimensional sample image can be determined, and according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, differentiable rendering can be used. The processor performs a rendering operation to obtain differentiable rendering information corresponding to the target object, where the differentiable rendering information may include rendering a segmentation mask, rendering a two-dimensional image, and rendering a depth image.

In step S13, according to the 2D sample image, the predicted segmentation mask, the depth image corresponding to the 2D sample image, and the differentiable rendering information, determine the total loss of self-supervised training of the pose prediction network .

For example, visual consistency constraints and geometric consistency constraints can be obtained by performing visual consistency constraints and geometric consistency constraints through the two-dimensional sample image, the predicted segmentation mask, and the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information obtained by rendering the predicted pose information. Self-supervised training total loss for pose-trained networks.

In step S14, the pose prediction network is trained according to the self-supervised training total loss.

For example, the parameters of the attitude prediction network can be adjusted according to the total loss of self-supervised training, until the total loss of self-supervised training meets the training requirements, and the self-supervised training of the attitude prediction network is completed.

In this way, the two-dimensional sample image can be predicted through a posture prediction network, and the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted posture information corresponding to the target object can be obtained, and the predicted posture information includes three-dimensional rotation. information and three-dimensional translation information, and perform a differentiable rendering operation according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, and obtain the differentiable rendering information corresponding to the target object. According to the 2D sample image, the predicted segmentation mask, the depth image corresponding to the 2D sample image, and the differentiable rendering information, determine the total training loss of the pose prediction network, and according to the self-supervised The training total loss trains the pose prediction network. According to the network training method provided by the embodiments of the present disclosure, the attitude prediction network can be trained by self-supervised two-dimensional sample images and depth images without label information, which can improve the training efficiency of the attitude prediction network while improving the accuracy of the attitude prediction network. .

In a possible implementation manner, the differentiable rendering information corresponding to the target object may include: rendering a segmentation mask, rendering a two-dimensional image, and rendering a depth image, and the segmentation according to the two-dimensional sample image, the predicted segmentation The mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information, to determine the total training loss of the pose prediction network, may include:

determining a first self-supervised training loss according to the two-dimensional sample image and the rendered two-dimensional image;

determining a second self-supervised training loss according to the predicted segmentation mask and the rendered segmentation mask;

determining a third self-supervised training loss according to the depth image corresponding to the two-dimensional sample image and the rendered depth image;

According to the first self-supervised training loss, the second self-supervised training loss and the third self-supervised training loss, a total self-supervised training loss of the pose prediction network is determined.

For example, the above-mentioned rendering segmentation mask may be a rendered mask image, and the pixel value of any pixel in the rendering segmentation mask is used to identify whether the pixel is a pixel in the target object in the sample two-dimensional image. . The rendered two-dimensional image can be a two-dimensional image obtained through the three-dimensional model of the target object and the predicted attitude information, and the rendered depth image can be a depth image obtained through the three-dimensional model of the target object and the predicted attitude information. The rendering process can be rendered through related technologies. This is done using a processor (for example, a Soft-Rasterizer), which is not repeated in this embodiment of the present disclosure.

The visual consistency constraint between the two-dimensional sample image and the rendered two-dimensional image can be established, the visual consistency constraint between the predicted segmentation mask and the rendered segmentation mask can be established, and the depth image corresponding to the two-dimensional sample image can be established. The geometric consistency constraints between the rendered depth images are described, and the pose prediction network is optimized by optimizing the two self-supervised constraints of visual consistency and geometric consistency.

The total training loss of the pose prediction network includes the loss determined by the visual consistency constraint and the loss determined by the geometric consistency constraint, wherein the loss determined by the visual consistency constraint includes the first training loss and the second training loss, and the geometric consistency is determined by the loss. The loss determined by the sexual constraint includes the third training loss, and the total loss of self-supervised training of the pose prediction network can be determined by the following formula (1).

L _self =L _visual +ηL _geom formula (1)

Among them, L _self represents the total loss of self-supervised training, L _visual represents the loss determined by the visual consistency constraint, and L _geom represents the loss determined by the geometric consistency constraint, that is, L _visual = the first self-supervised training loss + the second self Supervised training loss, L _geom = third self-supervised training loss, η represents the weight of the third self-supervised training loss.

In a possible implementation manner, the determining the first self-supervised training loss according to the two-dimensional sample image and the rendered two-dimensional image may include:

After the two-dimensional sample image and the rendered two-dimensional image are respectively converted into the color model LAB mode, according to the two-dimensional sample image after the conversion mode, the rendered two-dimensional image after the conversion mode, and the predicted segmentation mask, using the first loss function determines the first image loss;

According to the two-dimensional sample image, the rendered two-dimensional image and the predicted segmentation mask, a second loss function is used to determine a second image loss, and the second loss function is a loss function based on a multi-scale structural similarity index ;

According to the 2D sample image, the rendered 2D image and the predicted segmentation mask, a third loss function is used to determine a third image loss, and the third loss function is a multi-scale feature based on a deep convolutional neural network The loss function of distance;

The first self-supervised training loss is determined from the first image loss, the second image loss, and the third image loss.

For example, three loss functions can be employed to determine the first self-supervised training loss.

The first loss function is to convert the two-dimensional sample image and the rendered two-dimensional image to LAB (CIELab colormodel, LAB color model) mode respectively, and discard the luminance L channel of the two-dimensional sample image and rendered two-dimensional image after conversion to LAB mode. Then, the 1-norm distance between the two is calculated as the first image loss, and the first loss function can refer to the following formula (2).

表示预测分割掩码中第j个像素点。Among them, _Lab can represent the first image loss, M ^p represents the predicted segmentation mask, N ₊ can represent the region where the pixel value is greater than 0 in the predicted segmentation mask, ρ can represent the color space transformation operation, and ^IS can represent the two-dimensional sample image, IR can ^represent rendering a two-dimensional image,

Represents the jth pixel in the predicted segmentation mask.

The second loss function is a loss function based on MS-SSIM (Multi-Scale-Structural Similarity Index, multi-scale similarity index). The second loss function can refer to the following formula (3).

L _ms-ssim =1-ms-ssim(I ^S ⊙M ^P ,I ^R ,S) Formula (3)

Wherein, L _ms-ssim can represent the second image loss, ms-ssim represents a multi-scale similarity index function, ⊙ represents element-wise multiplication, and S is the number of scales used, exemplarily, S can take a value of 5.

The third loss function is a perceptual metric loss function based on a deep convolutional neural network. The pre-trained deep convolutional neural network can be used to extract the features of different layers of the two-dimensional sample image and render the two-dimensional image, and solve the two-dimensional sample image and rendering. The average 2-norm distance between the normalized features of the two-dimensional image is used as the third image loss, and the third loss function can refer to the following formula (4).

可以表示归一化的特征，N^l是第_l层特征的集合，|N^l|是第_l层特征的个数，示例性的，L可以取值为5。Among them, L _perceptual represents the third image loss, L is the total number of layers of the acquired features, _l can represent the layer number,

It can represent a normalized feature, N ^l is a set of features of the _lth layer, |N ^l | is the number of features of the _lth layer, exemplarily, L can take a value of 5.

The first self-supervised training loss is obtained by the weighted summation of the first image loss, the second image loss, and the third image loss.

In a possible implementation manner, the above-mentioned determination of the second self-supervised training loss according to the prediction segmentation mask and the rendering segmentation mask may include:

Based on the predicted segmentation mask and the rendered segmentation mask, a cross-entropy loss function is used to determine a second self-supervised training loss.

For example, due to the defect of the prediction segmentation mask, the consistency constraint between the prediction segmentation mask and the rendering segmentation mask adopts a cross-entropy loss function that readjusts the weights of the positive and negative regions, and can refer to the following formula (5) .

表示渲染分割掩码中第j个像素点。where L _mask represents the second self-supervised training loss, ^MR represents the rendering segmentation mask, N _- can represent the region where the pixel value is equal to 0 in the predicted segmentation mask,

Represents the jth pixel in the rendered segmentation mask.

In a possible implementation manner, the determining the third self-supervised training loss according to the depth image corresponding to the two-dimensional sample image and the rendered depth image may include:

Perform a back-projection operation on the depth image corresponding to the two-dimensional sample image and the rendered depth image, respectively, to obtain point cloud information corresponding to the depth image and point cloud information corresponding to the rendered depth image;

A third self-supervised training loss is determined according to the point cloud information corresponding to the depth image and the point cloud information corresponding to the rendered depth image.

For example, the depth image corresponding to the two-dimensional sample image and the rendered depth image can be respectively converted into point cloud information in the camera coordinate system through the back-projection operation, and the depth image corresponding to the two-dimensional sample image and the point corresponding to the rendered depth image can be converted into point cloud information in the camera coordinate system. The cloud information establishes a geometric consistency constraint, exemplarily, the geometric consistency constraint is established by the chamfer distance between the point cloud information. The back projection operation may refer to the following formula (6), and the calculation of the chamfer distance may refer to the formula (7).

Among them, D can represent the depth image (the depth image corresponding to the two-dimensional sample image or the rendered depth image), M can represent the segmentation mask (prediction segmentation mask or rendering segmentation mask), K can represent the camera internal parameters, x _j and y _j can represent the two-dimensional coordinates of the jth pixel.

Among them, p ^S can represent the point cloud information corresponding to the depth image corresponding to the two-dimensional sample image, p ^R can represent the point cloud information corresponding to the rendered depth image, and L _geom can represent the third self-supervised training loss.

That is, the total network loss can be calculated by the following formula (8):

L _self =L _mask +αL _ab +βL _ms-ssin +γL _perceptual +ηL _geom formula (8)

Among them, α, β, and γ are the weights of the first image loss, the second image loss, and the third image loss, for example: α=0.2, β=1, and γ=0.15.

After obtaining the total loss of self-supervised network training, the attitude prediction network can be trained according to the total loss of self-supervised network training. For an exemplary self-supervised training process of the attitude prediction network, refer to FIG. 2 .

In a possible implementation manner, the posture prediction network includes: a category prediction sub-network, a bounding box prediction sub-network, and a posture prediction sub-network,

The predicting the two-dimensional sample image through the attitude prediction network to obtain the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted attitude information corresponding to the target object may include:

Predict the two-dimensional sample image through the category prediction sub-network, and obtain category information corresponding to the target object in the two-dimensional sample image;

Predict the two-dimensional sample image through the bounding box prediction sub-network, and obtain bounding box information corresponding to the target object in the two-dimensional sample image;

The 2D sample image, the category information and the bounding box information are processed by the pose prediction sub-network to obtain the predicted segmentation mask corresponding to the target object in the 2D sample image and the corresponding target object the predicted pose information.

For example, the category prediction sub-network and the bounding box prediction sub-network can be constructed on the detector based on FPN (Feature Pyramid Network, Feature Pyramid). The category information of the target object is predicted, and the two-dimensional sample image is predicted according to the bounding box prediction sub-network, and the bounding box information corresponding to the target object in the two-dimensional sample image is obtained. The FPN features extracted by the detector are fused. Exemplarily, the features of different FPN layers can be reduced from 128 dimensions to 64 dimensions by 1×1 convolution, and then the spatial size of the features of different layers can be increased by bilinear interpolation. Sampling or down-sampling to 1/8 of the input image (if the input image is 480×640, it is unified to 60×80), and then the features of different layers of the same size are spliced in dimension.

After the FPN features are fused, new features are obtained by splicing the fused FPN features with the two-dimensional sample image and the two-dimensional coordinates corresponding to the two-dimensional sample image. The new features and bounding box information are obtained through the target detection special layer (ROI-Align) to obtain the features based on each target object, and the pose prediction sub-network processes the features of each target object to obtain the corresponding pose information and Predicted segmentation mask.

The pose prediction sub-network may include: a mask prediction sub-network, a quaternion sub-network, a 2D center point prediction sub-network and a center point distance prediction sub-network, a mask prediction sub-network outputs a prediction segmentation mask, and a quaternion sub-network The network outputs three-dimensional rotation information, and the 2D center point prediction sub-network outputs two-dimensional coordinates. The two-dimensional coordinates are transformed with the coordinate information output by the center point from the camera distance prediction sub-network to jointly obtain the three-dimensional translation information of the target object. The three-dimensional translation The information and the three-dimensional rotation information constitute the attitude information of the target object, refer to FIG. 3 .

In a possible implementation manner, before the two-dimensional sample image is predicted by the pose prediction network, the method may further include:

Rendering and synthesizing operations are performed according to the 3D model of the object and the preset posture information to obtain a synthesized 2D image and annotation information of the synthesized 2D image, where the annotation information of the synthesized 2D image includes annotated object category information and annotated bounding box information , preset pose information and preset synthetic segmentation mask;

Predict the synthetic 2D image through the pose prediction network to obtain prediction information of the synthetic 2D image, where the prediction information includes predicted object category information, predicted bounding box information, predicted synthetic segmentation mask, and predicted synthetic attitude information;

The pose training network is trained according to the prediction information and the annotation information of the synthesized two-dimensional image.

For example, the pose prediction network can be pre-trained on synthetic 2D images before self-supervised training of the pose prediction network with 2D sample information.

Exemplarily, a synthetic two-dimensional image can be obtained through OpenGL (OpenGraphics Library, Open Graphics Library) and a renderer based on a physics engine through the known three-dimensional model of the object and the preset attitude information, and a synthetic two-dimensional image can be obtained during the synthesis process. Annotation information of an image, including annotated object category information, annotated bounding box information, preset pose information, and preset synthetic segmentation mask.

The synthetic 2D image is processed by the pose prediction network to obtain the prediction information of the synthetic 2D image, the prediction information may include predicted object category information, predicted bounding box information, predicted synthesized segmentation mask and predicted synthesized pose information.

In the training process, the first loss can be calculated according to the predicted object category information and the labeled object category information, the second loss can be calculated according to the predicted bounding box information and the labeled bounding box information, and the predicted synthetic segmentation mask can be pre- and the preset synthetic segmentation mask. Calculate the third loss, and calculate the fourth loss according to the predicted synthetic attitude information and the preset attitude information. The total loss of the attitude prediction network can include the first loss, the second loss, the third loss and the fourth loss, which can be calculated by the following formula ( 9) Determine the total loss of the pose prediction network.

L _synthetic =λ _class L _focal +λ _box L _giou +λ _mask L _bce +λ _pose L _pose formula (9)

表示物体的三维模型M的点x经过预测姿态信息

和预设姿态信息

变换后的点之间的平均1范数距离，其中，

是预测姿态信息中的三维旋转信息，

是预测姿态信息中的三维平移信息，

是预设姿态信息中的三维旋转信息，

是预设姿态信息中的三维平移信息，λ_class、λ_box、λ_mask、λ_pose分别用于表示第一损失、第二损失、第三损失及第四损失的权重，可以取相同的值也可以取不同的值。L _synthetic can represent the total loss of the pose prediction network, L _focal , L _giou , L _bce , and L _pose represent the first loss, the second loss, the third loss and the fourth loss, respectively, where,

The point x representing the three-dimensional model M of the object has the predicted pose information

and preset attitude information

Average 1-norm distance between transformed points, where,

is the three-dimensional rotation information in the predicted attitude information,

is the three-dimensional translation information in the predicted attitude information,

is the three-dimensional rotation information in the preset attitude information,

is the three-dimensional translation information in the preset pose information. λ _class , λ _box , λ _mask , and λ _pose are used to represent the weights of the first loss, the second loss, the third loss and the fourth loss, respectively, and can take the same value or Can take different values.

After the pre-training based on the synthetic two-dimensional images, when using the two-dimensional sample images to self-supervised the training of the attitude prediction network, only the attitude prediction sub-network can be trained, and the network parameters of other networks will not be updated.

In order to make those skilled in the art better understand the embodiments of the present disclosure, the following describes the embodiments of the present disclosure through specific examples.

Referring to Figure 4, the training of the pose prediction network is divided into two stages.

In the first stage, the 3D model of the object is used to generate a large number of synthetic 2D images through OpenGL and the rendering method based on the physics engine. During the synthesis process, the identification information of the synthesized 2D image can be obtained, the pose prediction network is trained, and the category information and boundary of the object are output. Box information, predicted segmentation mask and pose information.

In the second stage, the unlabeled real 2D sample image is used to input the pose prediction network to obtain the predicted pose information and predicted segmentation mask of the target object in the 2D sample image, and the predicted pose information and the 3D model of the target object are input to the Differentiated renderer, obtains the rendered segmentation mask, renders the 2D image and renders the depth image, establishes visual consistency constraints between the rendered segmentation mask and the predicted segmentation mask, the rendered 2D image and the real 2D sample image, in the A geometric consistency constraint is established between the point cloud information corresponding to the depth image corresponding to the rendered depth map and the two-dimensional sample image, and the pose prediction network is trained by self-supervision by optimizing these two self-supervision constraints.

Embodiments of the present disclosure provide a gesture prediction method, including:

The image to be processed is predicted and processed by the attitude prediction network, and the attitude information of the target object in the image to be processed is obtained,

Wherein, the posture prediction network is obtained by training using any one of the network training methods described above.

For example, the pose prediction network trained by any of the foregoing methods can perform prediction processing on the image to be processed to obtain the pose information of the target object in the image to be processed.

In this way, according to the gesture prediction method provided by the embodiments of the present disclosure, the accuracy of gesture prediction can be improved.

It can be understood that the above-mentioned method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. Those skilled in the art can understand that, in the above method of the specific embodiment, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present disclosure also provides a network training device, an attitude prediction device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any of the network training methods and attitude prediction methods provided by the present disclosure, and the corresponding technical solutions and The description and reference to the corresponding records in the method section will not be repeated.

FIG. 5 shows a block diagram of a network training apparatus according to an embodiment of the present disclosure. As shown in FIG. 5 , the apparatus includes:

The prediction module 51 can be used to predict the two-dimensional sample image through a posture prediction network, and obtain the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted posture information corresponding to the target object. The information includes three-dimensional rotation information and three-dimensional translation information;

The rendering module 52 can be configured to perform a differentiable rendering operation according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, and obtain differentiable rendering information corresponding to the target object;

The determination module 53 can be used to determine the self-supervised training of the pose prediction network according to the two-dimensional sample image, the predicted segmentation mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information total loss;

A self-supervised training module 54 may be used to train the pose prediction network according to the self-supervised training total loss.

In this way, the two-dimensional sample image can be predicted through a posture prediction network, and the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted posture information corresponding to the target object can be obtained, and the predicted posture information includes three-dimensional rotation. information and three-dimensional translation information, and perform a differentiable rendering operation according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, and obtain the differentiable rendering information corresponding to the target object. According to the 2D sample image, the predicted segmentation mask, the depth image corresponding to the 2D sample image, and the differentiable rendering information, determine the total loss of self-supervised training of the pose prediction network, and according to the Self-supervised training total loss trains the pose prediction network. According to the network training device provided by the embodiments of the present disclosure, by self-supervised training of the attitude prediction network on two-dimensional sample images and depth images without label information, the accuracy of the attitude prediction network can be improved while the accuracy of the attitude prediction network can be improved. training efficiency.

In a possible implementation manner, the rendering information corresponding to the target object includes: rendering a segmentation mask, rendering a two-dimensional image, and rendering a depth image, and the determining module 53 can also be used for:

determining a first self-supervised training loss according to the two-dimensional sample image and the rendered two-dimensional image;

determining a second self-supervised training loss according to the predicted segmentation mask and the rendered segmentation mask;

determining a third self-supervised training loss according to the depth image corresponding to the two-dimensional sample image and the rendered depth image;

According to the first self-supervised training loss, the second self-supervised training loss and the third self-supervised training loss, a total self-supervised training loss of the pose prediction network is determined.

In a possible implementation manner, the determining module 53 may also be used to:

After the two-dimensional sample image and the rendered two-dimensional image are respectively converted into the color model LAB mode, according to the two-dimensional sample image after the conversion mode, the rendered two-dimensional image after the conversion mode, and the predicted segmentation mask, using the first loss function determines the first image loss;

According to the two-dimensional sample image, the rendered two-dimensional image and the predicted segmentation mask, a second loss function is used to determine a second image loss, and the second loss function is a loss function based on a multi-scale structural similarity index ;

According to the 2D sample image, the rendered 2D image and the predicted segmentation mask, a third loss function is used to determine a third image loss, and the third loss function is a multi-scale feature based on a deep convolutional neural network The loss function of distance;

The first self-supervised training loss is determined from the first image loss, the second image loss, and the third image loss.

In a possible implementation manner, the determining module 53 may also be used to:

Based on the predicted segmentation mask and the rendered segmentation mask, a cross-entropy loss function is used to determine a second self-supervised training loss.

In a possible implementation manner, the determining module 53 may also be used to:

Perform a back-projection operation on the depth image corresponding to the two-dimensional sample image and the rendered depth image, respectively, to obtain point cloud information corresponding to the depth image and point cloud information corresponding to the rendered depth image;

A third self-supervised training loss is determined according to the point cloud information corresponding to the depth image and the point cloud information corresponding to the rendered depth image.

In a possible implementation manner, the posture prediction network may include: a category prediction sub-network, a bounding box prediction sub-network, and a posture prediction sub-network, and the prediction module 51 may also be used for:

Predict the two-dimensional sample image through the category prediction sub-network, and obtain category information corresponding to the target object in the two-dimensional sample image;

Predict the two-dimensional sample image through the bounding box prediction sub-network, and obtain bounding box information corresponding to the target object in the two-dimensional sample image;

The 2D sample image, the category information and the bounding box information are processed by the pose prediction sub-network to obtain the predicted segmentation mask corresponding to the target object in the 2D sample image and the corresponding target object the predicted pose information.

In a possible implementation manner, the apparatus may further include:

The pre-training module can be used to perform a rendering synthesis operation according to the 3D model of the object and the preset posture information, and obtain a synthesized 2D image and label information of the synthesized 2D image, where the label information of the synthesized 2D image includes annotated objects Category information, labeled bounding box information, preset pose information, and preset synthetic segmentation masks;

Predict the synthetic 2D image through the pose prediction network to obtain prediction information of the synthetic 2D image, where the prediction information includes predicted object category information, predicted bounding box information, predicted synthetic segmentation mask, and predicted synthetic attitude information;

The pose prediction network is trained according to the prediction information and the annotation information of the synthesized two-dimensional image.

According to an aspect of the present disclosure, a posture prediction apparatus is provided, and the apparatus may include:

The prediction module is used to perform prediction processing on the image to be processed through the attitude prediction network, and obtain the attitude information of the target object in the image to be processed,

Wherein, the posture prediction network is obtained by training using any one of the network training methods described above.

In this way, according to the posture prediction apparatus provided by the embodiments of the present disclosure, the accuracy of posture prediction can be improved.

In some embodiments, the functions or modules included in the apparatuses provided in the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments. For specific implementation, reference may be made to the descriptions of the above method embodiments. For brevity, here No longer.

Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

Embodiments of the present disclosure also provide a computer program product, including computer-readable codes. When the computer-readable codes are run on a device, the processor in the device executes the method for implementing the network training method and Instructions for the pose prediction method.

Embodiments of the present disclosure further provide another computer program product for storing computer-readable instructions, which, when executed, cause the computer to perform the operations of the network training method and the attitude prediction method provided by any of the foregoing embodiments.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 6 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. terminal.

6, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814 , and the communication component 816 .

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

Memory 804 is configured to store various types of data to support operation at electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. Memory 804 may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when electronic device 800 is in operating modes, such as calling mode, recording mode, and voice recognition mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of electronic device 800 . For example, the sensor assembly 814 can detect the on/off state of the electronic device 800, the relative positioning of the components, such as the display and the keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or one of the electronic device 800 Changes in the position of components, presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as wireless network (WiFi), second generation mobile communication technology (2G) or third generation mobile communication technology (3G), or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A programmed gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation is used to perform the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium, such as a memory 804 comprising computer program instructions executable by the processor 820 of the electronic device 800 to perform the above method is also provided.

FIG. 7 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 7, electronic device 1900 includes processing component 1922, which further includes one or more processors, and a memory resource represented by memory 1932 for storing instructions executable by processing component 1922, such as applications. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) interface 1958 . The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows Server ^™ ), a graphical user interface based operating system (Mac OSX ^™ ) introduced by Apple, a multi-user multi-process computer operating system ( Unix ^™ ), Free and Open Source Unix-like Operating System (Linux ^™ ), Open Source Unix-like Operating System (FreeBSD ^™ ) or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as memory 1932 comprising computer program instructions executable by processing component 1922 of electronic device 1900 to perform the above-described method.

The present disclosure may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for carrying out operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code, written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims (12)

1. a network training method, is characterized in that, comprises: The two-dimensional sample image is predicted by the attitude prediction network, and the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted posture information corresponding to the target object are obtained. The predicted posture information includes three-dimensional rotation information and three-dimensional translation information; Perform a differentiable rendering operation according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, to obtain differentiable rendering information corresponding to the target object; determining the total loss of self-supervised training of the pose prediction network according to the two-dimensional sample image, the predicted segmentation mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information; The pose prediction network is trained according to the self-supervised training total loss. 2. The method according to claim 1, wherein the differentiable rendering information corresponding to the target object comprises: rendering a segmentation mask, rendering a two-dimensional image, and rendering a depth image, Determining the total loss of self-supervised training of the attitude prediction network according to the two-dimensional sample image, the predicted segmentation mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information, including: determining a first self-supervised training loss according to the two-dimensional sample image and the rendered two-dimensional image; determining a second self-supervised training loss according to the predicted segmentation mask and the rendered segmentation mask; determining a third self-supervised training loss according to the depth image corresponding to the two-dimensional sample image and the rendered depth image; According to the first self-supervised training loss, the second self-supervised training loss and the third self-supervised training loss, a total self-supervised training loss of the pose prediction network is determined. 3. The method according to claim 2, wherein determining the first self-supervised training loss according to the two-dimensional sample image and the rendered two-dimensional image, comprising: After the two-dimensional sample image and the rendered two-dimensional image are respectively converted into the color model LAB mode, according to the two-dimensional sample image after the conversion mode, the rendered two-dimensional image after the conversion mode, and the predicted segmentation mask, using the first loss function determines the first image loss; According to the two-dimensional sample image, the rendered two-dimensional image and the predicted segmentation mask, a second loss function is used to determine a second image loss, and the second loss function is a loss function based on a multi-scale structural similarity index ; According to the 2D sample image, the rendered 2D image and the predicted segmentation mask, a third loss function is used to determine a third image loss, and the third loss function is a multi-scale feature based on a deep convolutional neural network loss function of distance; The first self-supervised training loss is determined from the first image loss, the second image loss, and the third image loss. 4. The method according to claim 2 or 3, wherein determining the second self-supervised training loss according to the prediction segmentation mask and the rendering segmentation mask, comprising: Based on the predicted segmentation mask and the rendered segmentation mask, a cross-entropy loss function is used to determine a second self-supervised training loss. 5. The method according to any one of claims 2 to 4, wherein determining a third self-supervised training loss according to the depth image corresponding to the two-dimensional sample image and the rendered depth image, comprising: : Perform a back-projection operation on the depth image corresponding to the two-dimensional sample image and the rendered depth image, respectively, to obtain point cloud information corresponding to the depth image and point cloud information corresponding to the rendered depth image; A third self-supervised training loss is determined according to the point cloud information corresponding to the depth image and the point cloud information corresponding to the rendered depth image. 6. The method according to any one of claims 1 to 5, wherein the posture prediction network comprises: a category prediction sub-network, a bounding box prediction sub-network, and a posture prediction sub-network, The two-dimensional sample image is predicted through the attitude prediction network, and the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted attitude information corresponding to the target object are obtained, including: Predict the two-dimensional sample image through the category prediction sub-network, and obtain category information corresponding to the target object in the two-dimensional sample image; Predict the two-dimensional sample image through the bounding box prediction sub-network, and obtain bounding box information corresponding to the target object in the two-dimensional sample image; The 2D sample image, the category information and the bounding box information are processed by the pose prediction sub-network to obtain the predicted segmentation mask corresponding to the target object in the 2D sample image and the corresponding target object the predicted pose information. 7. The method according to claim 6, characterized in that, before the prediction of the two-dimensional sample image by the pose prediction network, the method further comprises: Rendering and synthesizing operations are performed according to the 3D model of the object and the preset posture information to obtain a synthesized 2D image and annotation information of the synthesized 2D image, where the annotation information of the synthesized 2D image includes annotated object category information and annotated bounding box information , preset pose information and preset synthetic segmentation mask; Predict the synthetic 2D image through the pose prediction network to obtain prediction information of the synthetic 2D image, where the prediction information includes predicted object category information, predicted bounding box information, predicted synthetic segmentation mask, and predicted synthetic attitude information; The pose prediction network is trained according to the prediction information and the annotation information of the synthesized two-dimensional image. 8. A gesture prediction method, wherein the method comprises: The image to be processed is predicted and processed by the attitude prediction network, and the attitude information of the target object in the image to be processed is obtained, Wherein, the posture prediction network is obtained by using the network training method described in any one of claims 1 to 7. 9. A network training device, comprising: The prediction module is used to predict the two-dimensional sample image through the attitude prediction network, and obtain the predicted segmentation mask corresponding to the target object in the two-dimensional sample image and the predicted attitude information corresponding to the target object, and the predicted attitude information includes 3D rotation information and 3D translation information; a rendering module, configured to perform a differentiable rendering operation according to the predicted posture information corresponding to the target object and the three-dimensional model corresponding to the target object, and obtain differentiable rendering information corresponding to the target object; A determination module, configured to determine the total loss of self-supervised training of the attitude prediction network according to the two-dimensional sample image, the predicted segmentation mask, the depth image corresponding to the two-dimensional sample image, and the differentiable rendering information ; A self-supervised training module for training the pose prediction network according to the self-supervised training total loss. 10. A posture prediction device, wherein the device comprises: The prediction module is used to perform prediction processing on the image to be processed through the attitude prediction network to obtain the attitude information of the target object in the image to be processed, Wherein, the posture prediction network is obtained by using the network training method described in any one of claims 1 to 7. 11. An electronic device, characterized in that, comprising: processor; memory for storing processor-executable instructions; wherein the processor is configured to invoke the memory-stored instructions to perform the method of any one of claims 1-8. 12. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the method of any one of claims 1 to 8 when executed by a processor.

Images