CN117975165B - Transparent object grabbing method based on depth complement - Google Patents

Abstract

The invention provides a transparent object grabbing method based on depth complementation, which comprises the following steps: identifying transparent objects in the RGB picture, and obtaining a depth map with a mask; performing feature extraction on the RGB picture and the depth map with the mask by using a teacher network to obtain global features and local features of the transparent object; wherein, the teacher network is constructed based on a transducer network; based on the composite distillation loss function, learning global features and local features by using a student network, inputting the RGB picture and the depth map with the mask into the learned student network for association processing, and obtaining a predicted depth map; wherein, student network builds based on CNN network. According to the invention, a student network with high precision, high efficiency and high robustness is taught by using a teacher network, and a teacher network with slightly lower precision or a student network with slightly lower precision and higher efficiency is selected according to actual needs to be deployed in a real robot so as to realize high-precision grabbing.

Description

A transparent object grasping method based on depth completion

Technical Field

The present invention belongs to the technical field of transparent object grasping, and in particular relates to a transparent object grasping method based on depth completion.

Background Art

Transparent objects are common objects in our daily life, such as glasses, glasses, plastic bottles, etc. In addition, there are many transparent objects in the chemical industry, manufacturing industry, biological industry, etc. However, due to the special properties of transparent objects, such as refraction and reflection, it is difficult for standard depth cameras to capture their accurate depth information. For example, Microsoft's Kinect depth camera can irradiate the laser onto the surface of the object, and then capture the dot matrix through the CMOS sensor to calculate the shape of the object. This method has a good recognition effect for non-transparent objects. However, for transparent objects, part of the laser will penetrate the transparent object and refract to the background, so that the camera can obtain the depth information of the background when capturing; part of the laser will be reflected on the surface of the object, and the camera cannot obtain its corresponding depth information at this time; the remaining part of the laser hits the surface of the object, and this part of the laser will be captured by the camera and generate the corresponding depth. However, due to these drift points and lost points, the accuracy of the robot's grasping frame prediction is greatly reduced, making it impossible to accurately grasp.

In order to capture the depth information of transparent objects, existing methods are mainly divided into the following four categories:

A method based on multi-view annotation and key point estimation. By clamping the depth camera and binocular camera on the robot arm and continuously shooting along a given trajectory, a multi-view dataset is obtained, and then machine learning and other algorithms are used to extract the key points of the object for depth completion.

The method based on object segmentation and two-dimensional edge analysis extracts the rough outline of the transparent object through the two-dimensional plane information of the transparent object, but cannot further perceive the depth information of the transparent object.

Methods based on known transparent object models: The geometric shape of a transparent object is estimated by using a known 3D model of the object. This method requires knowing the 3D model of the transparent object in advance, and its use conditions are restricted.

A deep learning-based method that learns mapping information from 2D images and depth maps by training the network, and then transfers it to transparent objects for depth completion.

Current implementation:

Fang et al. proposed the TransCG method, which concatenates the RGB image and the depth map, then uses multiple convolutional layers (convolution-dense convolution-convolution-downsampling) to downsample and extract features, and then uses multiple upsampling layers (convolution-dense convolution-convolution-upsampling) to restore the image. However, this method uniformly processes two different types of features and cannot extract the mapping relationship; in addition, the application of multiple dense convolutions increases the computational overhead.

Shao et al. proposed the URCDC method, which reconstructs the depth map of outdoor scenes by inputting RGB images into a Transformer network and a CNN network respectively, and then uses distillation learning to make the depth map predicted by the student close to the depth map predicted by the teacher, thereby outputting the student's results. Although this method uses distillation learning to improve the efficiency of prediction, the input of this method is an RGB image, and the network cannot extract the mapping relationship between two different features when extracting features, which weakens the learning effect; in addition, this method relies on the color texture information of outdoor scenes, and is not effective for transparent object scenes that lack texture.

In summary, the shortcomings of the prior art are:

It needs to rely on manual data collection, which has high labor and time costs;

The method of using multiple sensors to collect data leads to higher hardware costs;

Existing depth camera hardware cannot collect accurate depth information of transparent objects. The collected results contain lost points and drift points, and the collection results are not ideal.

Using a single neural network model based on Transformer architecture and dense convolutional architecture is too computationally expensive to be deployed in real scenarios.

Therefore, it is of great significance to propose a robot grasping frame that can accurately predict the transparent object and can adapt to various complex environments and backgrounds.

Summary of the invention

The purpose of the present invention is to provide a transparent object grasping method based on depth completion, which uses a teacher network to teach a student network with high precision, high efficiency and high robustness, and selects a teacher network with high precision and slightly lower efficiency or a student network with slightly lower precision and higher efficiency according to actual needs and deploys them in a real robot to achieve high-precision grasping.

To achieve the above object, the present invention provides a transparent object grasping method based on depth completion, comprising:

Identify transparent objects in RGB images and obtain masked depth maps;

Using a teacher network to extract features from the RGB image and the masked depth map to obtain global features and local features of the transparent object; wherein the teacher network is constructed based on a Transformer network;

Based on the composite distillation loss function, the global features and local features are learned using a student network, and the RGB image and the masked depth map are input into the learned student network for association processing to obtain a predicted depth map; wherein the student network is constructed based on a CNN network;

Inputting the predicted depth map into the GR-CNN network, obtaining a grabbing frame of the transparent object, and completing the grabbing of the transparent object based on the grabbing frame;

Identify transparent objects in RGB images and obtain a masked depth map including:

Using the Deeplabv3+ algorithm to identify transparent objects in the RGB image and extract transparent object masks;

Based on the transparent object mask, remove the corresponding transparent object area in the depth map by mapping the position information to obtain the masked depth map;

Inputting the RGB image and the masked depth map into the learned student network for association processing includes:

Input the RGB image and the masked depth map into the convolution layer respectively for preliminary feature extraction, and obtain the RGB image features and depth map features after preliminary extraction;

The depth map features after preliminary extraction are processed by downsampling layer for dimensionality reduction; each depth map feature after dimensionality reduction is then sent to a downsampling layer for dimensionality reduction;

The RGB image features and depth map features after preliminary extraction are input into the first consistent feature association module for feature association processing; starting from the second feature association module, the output processed by the previous feature association module and the depth map features after the previous dimensionality reduction processing through the downsampling layer are used as the input of the next consistent feature association module, and so on, and are continuously input into three consistent feature association modules for further feature association; wherein, the size of the associated features output by each consistent feature association module is half of the output size of the previous consistent feature association module, but the dimension is increased to twice the original; and the size of the depth map features is also reduced to half of the previous layer after passing through the downsampling layer, and the dimension is increased to twice the original; after the feature association processing is performed by four consistent feature association modules, the encoded second integrated features are finally obtained;

Using a second decoder, decoding the second integrated features to obtain a predicted depth map;

The initially extracted RGB image features and depth map features are input into the first consistent feature association module, and feature association processing includes:

The initially extracted RGB image features and depth map features are respectively input into the convolutional layer for channel compression;

After concatenating the compressed depth map features with the corresponding uncompressed RGB image features in terms of the number of channels, a convolution operation is performed to obtain the similarity weight of the depth map features;

Multiplying the similarity weight of the depth map feature by the uncompressed depth map feature bit by bit to obtain a reliable depth map position feature;

After concatenating the compressed RGB image features with the corresponding uncompressed depth image features in terms of the number of channels, a convolution operation is performed to obtain the similarity weights of the RGB image features;

Multiply the similarity weight of the RGB image feature by the uncompressed RGB image feature bit by bit to obtain a reliable RGB image position feature;

The reliable depth map position features and the reliable RGB image position features are summed by bitwise addition to obtain a final reliable position feature map;

For the final reliable position feature map, linear rectification function ReLU, batch normalization BatchNormalization and convolution are used to perform feature association to obtain associated features.

Optionally, using a teacher network to extract features from the RGB image and the masked depth map includes:

Performing block embedding processing on the RGB picture and the masked depth map respectively;

The depth map after block embedding is processed by three downsampling layers in turn for dimensionality reduction. The depth map after dimensionality reduction in each downsampling layer is used as the input of a position association module.

The RGB image after block embedding processing, the depth map after block embedding processing, and the depth map after the depth map after block embedding processing are encoded by a position association module to obtain the encoded first integrated feature; wherein the position association module includes 4, the input of the first position association module is the RGB image after block embedding processing and the depth map after block embedding processing, and the input of the subsequent position association module is the output of the previous position association module and the depth map after the depth map after block embedding processing is reduced by the corresponding downsampling layer;

The first decoder is used to perform feature decoding on the encoded first integrated feature to obtain the global feature and the local feature of the transparent object.

Optionally, encoding the block-embedded RGB picture and the block-embedded depth map through a first position association module includes:

Initialize the RGB image and the depth map using layer normalization to obtain RGB image features and depth map features, use the RGB image features as a search request Q, and gradually search the RGB image features and the depth map features to obtain key value elements;

Input the key-value elements into the multi-head attention mechanism module to obtain the confidence of the retrieval results;

Based on layer normalization and a multi-layer perceptron, obtaining correlation features between the RGB image features and the depth map features in the confidence;

Performing an offset window operation on the associated features and the depth map features to obtain a new retrieval result confidence;

Based on the new retrieval result confidence, obtaining the encoded integrated features;

The correlation feature between the RGB picture and the depth map is:

F _corr =MLP(LN(F _qkv ))+F _qkv

Among them, F _corr is the correlation feature, F _qkv is the result of multi-head attention calculation, LN is layer normalization, and MLP is a multi-layer perceptron.

Optionally, performing an offset window operation on the associated features and the depth map features includes:

Move each pixel in the associated features and the depth map features to the lower right corner by two pixels, and then re-block the associated features and the depth map features;

The block features are used as new search requests Q, and the index elements K and content elements V corresponding to the block-related features and depth map features are gradually searched to obtain new key-value elements;

The new key-value elements are input into the multi-head attention mechanism module to calculate the new retrieval result confidence.

Optionally, the composite distillation loss function is:

in, The loss function for the composite distillation is, is the distance loss, is the structural loss, is the difference between the real depth map and the student predicted depth map, is the difference between the predicted depth maps of the teacher and the student, and λ ₁ , λ ₂ , and λ ₃ are the constant weights of the corresponding losses.

The present invention has the following beneficial effects:

The present invention uses a teacher network to extract features from the RGB image and the masked depth map to obtain global and local features of transparent objects; wherein the teacher network is constructed based on the Transformer network; the present invention retains the advantages of using the transformer architecture to extract global and local features, trains a teacher network that currently leads all other transparent object depth completions, and teaches a student network with accuracy second only to the teacher network, but twice the speed, through distillation learning. In addition, by deploying in a real environment, the robustness of the student network is effectively verified, achieving high-precision capture.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

FIG1 is a schematic flow chart of a method for grasping a transparent object based on depth completion according to an embodiment of the present invention;

FIG2 is a schematic diagram of a location association module in a teacher network according to an embodiment of the present invention;

FIG3 is a schematic diagram of a NeWCRFs decoding module in a teacher network according to an embodiment of the present invention;

FIG4 is a schematic diagram of a consistent feature association module in a student network according to an embodiment of the present invention;

FIG5 is a schematic diagram of a composite distillation loss function according to an embodiment of the present invention;

FIG6 is a schematic diagram of eight types of common transparent objects according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of an experiment of a robot grasping a transparent object according to an embodiment of the present invention.

DETAILED DESCRIPTION

It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

As shown in Figures 1-5, this embodiment proposes a transparent object grasping method based on depth completion. Through the distillation learning algorithm for the existing data set, specifically, on the teacher network side, this embodiment proposes a network based on the Transformer architecture to associate RGB image features with the depth map to achieve position mapping between features. Thanks to the Transformer architecture, this embodiment can extract the global and local features of transparent objects, and finally output a high-precision depth map for teaching the student network; on the student network side, in order to ensure efficiency, this embodiment proposes a neural network based on the CNN architecture. The network uses a more efficient way to associate RGB and depth maps, and learns global and local features from the teacher network. It achieves twice the teacher network calculation speed at a slight sacrifice of accuracy, reaching a speed of 48 frames per second, and can be calculated in real time. In addition, through the application on the robot side, the robustness of the method of this embodiment is effectively verified.

A transparent object grasping method based on depth completion proposed in this embodiment specifically includes:

Identify transparent objects in RGB images and obtain masked depth maps;

Furthermore, obtaining a masked depth map includes:

Use the Deeplabv3 algorithm to identify transparent objects in RGB images and extract transparent object masks;

Based on the transparent object mask, the corresponding transparent object area in the depth map is removed through position information mapping to obtain a masked depth map.

Specifically, in this embodiment, the Deeplabv3 algorithm is first used to identify transparent objects in the RGB image, and the transparent object mask Mask is extracted. Then, the corresponding transparent object area in the depth map is dug out through position information mapping to prevent these uncertain points from affecting the model learning the mapping relationship between RGB and depth map. At this time, a masked depth map Mask Depth is obtained.

The teacher network is used to extract features from RGB images and masked depth images to obtain global and local features of transparent objects. The teacher network is built based on the Transformer network.

Furthermore, the teacher network is used to extract features from the RGB image and the masked depth map, including:

Performing block embedding processing on the RGB picture and the masked depth map respectively;

The depth map after block embedding is processed by three downsampling layers in turn for dimensionality reduction. The depth map after dimensionality reduction in each downsampling layer is used as the input of a position association module.

The RGB image after block embedding processing, the depth map after block embedding processing, and the depth map after the depth map after block embedding processing are encoded by a position association module to obtain the encoded first integrated feature; wherein the position association module includes 4, the input of the first position association module is the RGB image after block embedding processing and the depth map after block embedding processing, and the input of the subsequent position association module is the output of the previous position association module and the depth map after the depth map after block embedding processing is reduced by the corresponding downsampling layer;

The first decoder is used to perform feature decoding on the encoded first integrated feature to obtain the global feature and the local feature of the transparent object.

The block is embedded in the processed RGB picture and the block is embedded in the processed depth map, and encoding is performed by a first position association module, including:

Initialize the RGB image and the depth map using layer normalization to obtain RGB image features and depth map features, use the RGB image features as a search request Q, and gradually search the RGB image features and the depth map features to obtain key value elements;

Input the key-value elements into the multi-head attention mechanism module to obtain the confidence of the retrieval results;

Based on layer normalization and a multi-layer perceptron, obtaining correlation features between the RGB image features and the depth map features in the confidence;

Performing an offset window operation on the associated features and the depth map features to obtain a new retrieval result confidence;

Based on the new retrieval result confidence, obtaining the encoded integrated features;

Specifically, in this embodiment, the RGB image and the masked depth image Mask Depth are first subjected to Patch Embedding processing, and the image is divided into blocks to facilitate the extraction of fine local features. For the masked depth image, this embodiment uses a common downsampling layer for dimensionality reduction; and for the RGB image, this embodiment proposes a position correlation block (PCB). Specifically, this embodiment first uses layer normalization to initialize the RGB image and the depth image, and then uses the RGB feature as the retrieval request Q to retrieve the corresponding key-value pairs K and V from the RGB image and the depth image respectively. This embodiment inputs these three elements together into the multi-head attention mechanism module to calculate the confidence of the retrieval result, which includes the correlation features between the RGB image and the depth image. Then, this embodiment uses a layer normalization and a multi-layer perceptron to calculate the correlation matrix, which is expressed as follows:

Q＝ _xi ,K＝ _xi ,V＝ _xd

F _qkv =SoftMax(Q·K ^T +B)·V+I

F _corr =MLP(LN(F _qkv ))+E _qkv

Where I is the original image feature, LN is layer normalization, MLP is a multi-layer perceptron, _xi is the RGB image block feature, _xd is the depth block feature, _Fqkv is the result of multi-head attention calculation, and _Fcorr is the final correlation matrix feature.

However, since this feature only considers the association results of local blocks and does not include global features, the overall effect produced during completion is poor. To this end, this embodiment introduces the shifted window operation ShiftedWindows of Swin-Transformer. Specifically, by moving each pixel in all feature results to the lower right corner by two pixels, and then re-blocking the feature results, due to the offset operation, the existing block will contain the pixel features of the original adjacent area. Then this embodiment uses the same approach as above, and uses the offset feature as the retrieval request Q to retrieve the corresponding key-value pairs K and V from the feature map and the depth map after offset processing, and input these three elements together into the multi-head attention mechanism module to calculate the new retrieval result confidence. At this time, the features of this embodiment contain local and global association information. It is worth mentioning that after each position association module processing, the feature size dimension will become half of the original, and the number of channels will increase to twice the original. In order to extract multi-scale features, this embodiment superimposes 4 position association modules and obtains the encoded integrated feature Integrated Feature.

For the decoder, this embodiment uses a decoder based on the neural window fully connected conditional random field network NeWCRFs (Neural Window Fully-connected CRFs) for feature decoding. It is worth mentioning that when calculating the attention mechanism, the module uses the same RGB-associated depth map method as above for feature association, that is, using the RGB image feature as the retrieval request Q, and retrieving the corresponding key-value pairs K and V from the RGB image feature and the depth map feature respectively, but this time the integrated feature is used as Q for the retrieval request, and the equivalent key-value pairs K and V are searched in the integrated feature Q and the encoded feature V. Finally, after superimposing 4 layers of decoding layers, this embodiment obtains the predicted depth map of the teacher network end.

Based on the composite distillation loss function, the student network is used to learn global features and local features, and the RGB image and the masked depth map are input into the learned student network for association processing to obtain the predicted depth map; among them, the student network is built based on the CNN network.

Furthermore, the RGB image and the masked depth map are input into the learned student network for association processing including:

Input the RGB image and the masked depth map into the convolution layer respectively for preliminary feature extraction, and obtain the RGB image features and depth map features after preliminary extraction;

The depth map features after preliminary extraction are processed by downsampling layer for dimensionality reduction; each depth map feature after dimensionality reduction is then sent to a downsampling layer for dimensionality reduction;

The RGB image features and depth map features after preliminary extraction are input into the first consistent feature association module for feature association processing; starting from the second feature association module, the output processed by the previous feature association module and the depth map features after the previous dimensionality reduction processing through the downsampling layer are used as the input of the next consistent feature association module, and so on, and are continuously input into three consistent feature association modules for further feature association; wherein, the size of the associated features output by each consistent feature association module is half of the output size of the previous consistent feature association module, but the dimension is increased to twice the original; and the size of the depth map features is also reduced to half of the previous layer after passing through the downsampling layer, and the dimension is increased to twice the original; after the feature association processing is performed by four consistent feature association modules, the encoded second integrated features are finally obtained;

Using a second decoder, decoding the second integrated features to obtain a predicted depth map;

The initially extracted RGB image features and depth map features are input into the first consistent feature association module, and feature association processing includes:

The initially extracted RGB image features and depth map features are respectively input into the convolutional layer for channel compression;

After concatenating the compressed depth map features with the corresponding uncompressed RGB image features in terms of the number of channels, a convolution operation is performed to obtain the similarity weight of the depth map features;

Multiplying the similarity weight of the depth map feature by the uncompressed depth map feature bit by bit to obtain a reliable depth map position feature;

After concatenating the compressed RGB image features with the corresponding uncompressed depth image features in terms of the number of channels, a convolution operation is performed to obtain the similarity weights of the RGB image features;

Multiply the similarity weight of the RGB image feature by the uncompressed RGB image feature bit by bit to obtain a reliable RGB image position feature;

The reliable depth map position features and the reliable RGB image position features are summed by bitwise addition to obtain a final reliable position feature map;

For the final reliable position feature map, linear rectification function ReLU, batch normalization BatchNormalization and convolution are used to perform feature association to obtain associated features.

Specifically, in this embodiment, the RGB image and the masked depth image Mask Depth are first input into the convolutional network to extract primary features. For the depth image, this embodiment uses an ordinary downsampling layer for dimensionality reduction. For the RGB image, this embodiment proposes a consistent feature correlation module (CFCM). Specifically, this embodiment first inputs the RGB image and the masked depth image into the convolutional layer for preliminary feature extraction to obtain the RGB image features and depth image features after preliminary extraction. The downsampling layer is then used to perform dimensionality reduction processing on the depth image features after preliminary extraction. Each depth image feature after dimensionality reduction processing enters a downsampling layer for dimensionality reduction processing. Then the RGB image features and depth image features after preliminary extraction are input into the first consistent feature correlation module for feature correlation processing. Starting from the second feature correlation module, the output after processing by the previous feature correlation module is combined with the output of the previous feature correlation module through the downsampling layer. The depth map features after dimensionality reduction processing are used as the input of the next consistent feature association module, and so on, and are continuously input into three consistent feature association modules for further feature association; wherein, the size of the associated features output by each consistent feature association module is half of the output size of the previous consistent feature association module, but the dimension is increased to twice the original; and the size of the depth map features after passing through the downsampling layer is also reduced to half of the previous layer, while the dimension is increased to twice the original; after feature association processing is performed through four consistent feature association modules, the encoded second integrated features are finally obtained; finally, the second decoder is used to decode the second integrated features to obtain the predicted depth map.

Specifically, the initially extracted RGB image features and depth map features are input into the first consistent feature association module, and feature association processing includes:

The RGB image features and depth map features after preliminary extraction are respectively input into the convolution layer for channel number compression; the compressed depth map features are spliced with the corresponding uncompressed RGB image features in terms of the number of channels, and then a convolution operation is performed to obtain the similarity weight of the depth map features; the similarity weight of the depth map features is bitwise multiplied with the uncompressed depth map features to obtain the reliable depth map position features; the compressed RGB image features are spliced with the corresponding uncompressed depth map features in terms of the number of channels, and then a convolution operation is performed to obtain the similarity weight of the RGB image features; the weight reflects the degree of similarity between two different features, and the higher the similarity means the higher the degree of association, which also means the higher the mapping reliability; the similarity weight of the RGB image features is bitwise multiplied with the uncompressed RGB image features to obtain the reliable RGB image position features; the reliable depth map position features and the reliable RGB image position features are summed by bitwise addition to obtain the final reliable position feature map; for the final reliable position feature map, the linear rectification function ReLU, batch normalization and convolution are used to perform feature association to obtain the associated features. By stacking four consistent feature association modules, this embodiment can obtain the encoded integrated feature.

For the decoder, this embodiment uses a conventional upsampling module for depth restoration. Specifically, the upsampling module includes a convolution layer and an upsampling layer. The integrated features are upsampled and concatenated with the encoded features, and then output to the next layer for processing. By stacking four groups of upsampling modules, this embodiment can obtain the depth map predicted by the student network.

In order to prevent the student network from learning only the regional features of the teacher network, this embodiment designs a composite distillation loss function to ensure the learning effect, which includes distance loss, structure loss and edge loss. The three loss functions are defined as follows:

Distance loss: This embodiment first uses the scale-invariant Log loss function as the error calculation between prediction points, and its formula is as follows:

Where △E _gt represents the logarithmic loss between the student network and the teacher network, and △E _p represents the logarithmic loss between the student network and the teacher network. The formula is as follows:

ΔE _p =logD _s -logD _t ,ΔE _gt =logD _s -logD _gt

α and β are scaling constants, and λ is the variance minimization factor. These three constants are 3, 7, and 0.85 respectively.

Structural loss: This embodiment uses the structural parameter C in the structural similarity index measure (SSIM) to measure the structural loss, and the formula is as follows:

Where x and y refer to two sets of depth maps, σ _x represents the variance of the depth map, σ _x,y represents the covariance, and θ is a constant to prevent the denominator from being 0. In the structural loss, this embodiment considers two sets of structural differences, one is the difference between the real depth map and the depth map predicted by the teacher, and the other is the difference between the depth map predicted by the teacher and the depth map predicted by the student. Finally, this embodiment calculates the two sets of differences using the mean squared error (MSE), and the formula is as follows:

Where MSE is the mean square error.

Edge loss: Since the completion of the edge area of the object will produce a large change in depth value, mainly due to the difference between the depth value of the object surface and the background, this embodiment designs an edge loss, which mainly calculates the difference in the horizontal and vertical object edge positions. The definition is as follows:

in represents the coordinate point (x, y) in the depth map r, M _x represents the horizontal transparent area mask, and ⊙ represents bitwise multiplication. Similarly, the vertical difference The definition is as follows:

Where _My is the vertical transparent area mask. In this embodiment, two sets of difference values are calculated respectively, one is the difference between the real depth map and the student predicted depth map:

The other set is the difference between the predicted depth maps between the teacher and the student:

Finally, the distillation loss function of this embodiment is defined as follows:

Where λ ₁ , λ ₂ , and λ ₃ are set to 0.1, 0.3, and 0.7, respectively.

Through the distillation method, this embodiment can achieve the complete transfer of knowledge from the teacher network to the student network, and teach a student network with high accuracy, high speed and good robustness to perform depth completion of transparent objects.

By training the network, the teacher network and the student network can output complete depth maps respectively. In this embodiment, the prediction results are evaluated using point-by-point absolute error MAE (mean absolute error), point-by-point root mean square error RMSE (mean absolute error), absolute relative error REL (absolute relative difference) and the accuracy ratio δ of the predicted depth map to the true depth map. The accuracy ratio threshold is measured using the common three ratios of 1.05, 1.10 and 1.25, that is, if the error between the predicted depth map and the true depth map is less than 1.05, 1.10 and 1.12, the prediction is considered to be accurate, and the higher the prediction score, the better the prediction effect.

The completion accuracy ratio δ of the teacher network in synthetic objects can reach δ _1.05 ＝77.56, δ _1.10 ＝93.83 and δ _1.25 ＝99.68, reaching 0.021 in RMSE index, 0.035 in REL index and 0.018 in MAE index; the completion accuracy ratio δ of real objects can reach δ _1.05 ＝80.78, δ _1.10 ＝94.91 and δ _{1.25 ＝99.56, reaching 0.021 in RMSE index, 0.032 in REL index and 0.018 in MAE index, which is the best result achieved among all transparent object depth completion methods at present; while the completion accuracy ratio δ of the student network in synthetic objects is δ 1.05 ＝} 72.48, δ _1.10 ＝91.23 and δ _1.25 ＝99.68, reaching 0.021 in RMSE index, 0.032 in REL index and 0.018 in MAE index. =99.32, reaching 0.024 in RMSE, 0.040 in REL and 0.020 in MAE; the completion accuracy ratios δ of real objects are δ _1.05 =71.33, δ _1.10 =87.38 and δ _1.25 =98.91, reaching 0.032 in RMSE, 0.044 in REL and 0.025 in MAE, which is the second best result among all current transparent object depth completion methods. The data results are shown in Table 1 below:

Table 1

Among them, CG, DG, DFNet, and FDCT are the latest transparent object completion methods.

In terms of computing efficiency, the teacher network can reach 24 frames per second, which basically meets the requirements of common cameras; the student network can reach 48 frames per second, which meets the requirements of some high-speed cameras in industry. The data results are shown in Table 2 below:

Table 2

After obtaining the complete depth map, this embodiment inputs the depth map into the GR-CNN network to predict the object grasping points. The network uses multiple two-dimensional convolutions and residual blocks to extract features and predict the grasping frame of the output object. This embodiment inputs the grasping frame to the robot, and the robot generates a corresponding trajectory based on the inverse kinematics and grasps it. Among the given 8 types of common transparent glass cups, this embodiment performs 10 grasps on each object, and any grasp that takes more than 10 seconds is considered a success. In the experiment, the teacher network can achieve a grasping success rate of 83.75%, and the student network can achieve a grasping success rate of 78.75%. The robot grasping transparent object experiment is shown in Figures 6 and 7

This embodiment applies the distillation learning algorithm to transparent object grasping, which is the first method to use this approach.

Teacher network method,The current accuracy achieved by the teacher method exceeds other existing algorithms for,depth completion of transparent objects.

(1) This embodiment retains the advantages of using the transformer architecture to extract global and local features, trains an algorithm that currently leads all other transparent object depth completion algorithms, and teaches a student network that is second only to the teacher network in accuracy but twice as fast through distillation learning. In addition, by deploying it in a real environment, the robustness of the student network is effectively verified.

(2) Use existing data sets without spending time and manpower on data collection;

(3) No need to rely on complex and expensive depth cameras for data collection;

(4) By training or fine-tuning the model, grasping can be performed on a real robot.

(5) Depending on the actual situation, a teacher network with higher accuracy or a student network with higher efficiency can be selected for deployment.

The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (5)

1. A method for capturing transparent objects based on depth completion, comprising: Identify transparent objects in RGB images and obtain masked depth maps; Using a teacher network to extract features from the RGB image and the masked depth map to obtain global features and local features of the transparent object; wherein the teacher network is constructed based on a Transformer network; Based on the composite distillation loss function, the global features and local features are learned using a student network, and the RGB image and the masked depth map are input into the learned student network for association processing to obtain a predicted depth map; wherein the student network is constructed based on a CNN network; Inputting the predicted depth map into the GR-CNN network, obtaining a grabbing frame of the transparent object, and completing the grabbing of the transparent object based on the grabbing frame; Identify transparent objects in RGB images and obtain a masked depth map including: Using the Deeplabv3+ algorithm to identify transparent objects in the RGB image and extract transparent object masks; Based on the transparent object mask, remove the corresponding transparent object area in the depth map by mapping the position information to obtain the masked depth map; Inputting the RGB image and the masked depth map into the learned student network for association processing includes: Input the RGB image and the masked depth map into the convolution layer respectively for preliminary feature extraction, and obtain the RGB image features and depth map features after preliminary extraction; The depth map features after preliminary extraction are processed by downsampling layer for dimensionality reduction; each depth map feature after dimensionality reduction is then sent to a downsampling layer for dimensionality reduction; The RGB image features and depth map features after preliminary extraction are input into the first consistent feature association module for feature association processing; starting from the second feature association module, the output processed by the previous feature association module and the depth map features after the previous dimensionality reduction processing through the downsampling layer are used as the input of the next consistent feature association module, and so on, and are continuously input into three consistent feature association modules for further feature association; wherein, the size of the associated features output by each consistent feature association module is half of the output size of the previous consistent feature association module, but the dimension is increased to twice the original; and the size of the depth map features is also reduced to half of the previous layer after passing through the downsampling layer, and the dimension is increased to twice the original; after the feature association processing is performed by four consistent feature association modules, the encoded second integrated features are finally obtained; Using a second decoder, decoding the second integrated features to obtain a predicted depth map; The initially extracted RGB image features and depth map features are input into the first consistent feature association module, and feature association processing includes: The initially extracted RGB image features and depth map features are respectively input into the convolutional layer for channel compression; After concatenating the compressed depth map features with the corresponding uncompressed RGB image features in terms of the number of channels, a convolution operation is performed to obtain the similarity weight of the depth map features; Multiplying the similarity weight of the depth map feature by the uncompressed depth map feature bit by bit to obtain a reliable depth map position feature; After concatenating the compressed RGB image features with the corresponding uncompressed depth image features in terms of the number of channels, a convolution operation is performed to obtain the similarity weights of the RGB image features; Multiply the similarity weight of the RGB image feature by the uncompressed RGB image feature bit by bit to obtain a reliable RGB image position feature; The reliable depth map position features and the reliable RGB image position features are summed by bitwise addition to obtain a final reliable position feature map; For the final reliable position feature map, linear rectification function ReLU, batch normalization BatchNormalization and convolution are used to perform feature association to obtain associated features. 2. The method for capturing transparent objects based on depth completion according to claim 1, wherein the step of extracting features from the RGB image and the masked depth map using a teacher network comprises: Performing block embedding processing on the RGB picture and the masked depth map respectively; The depth map after block embedding is processed by three downsampling layers in turn for dimensionality reduction. The depth map after dimensionality reduction in each downsampling layer is used as the input of a position association module. The RGB image after block embedding processing, the depth map after block embedding processing, and the depth map after the depth map after block embedding processing are encoded by a position association module to obtain the encoded first integrated feature; wherein the position association module includes 4, the input of the first position association module is the RGB image after block embedding processing and the depth map after block embedding processing, and the input of the subsequent position association module is the output of the previous position association module and the depth map after the depth map after block embedding processing is reduced by the corresponding downsampling layer; The first decoder is used to perform feature decoding on the encoded first integrated feature to obtain the global feature and the local feature of the transparent object. 3. The method for capturing a transparent object based on depth completion according to claim 2, wherein encoding the block-embedded RGB image and the block-embedded depth map through a first position association module comprises: Initialize the RGB image and the depth map using layer normalization to obtain RGB image features and depth map features, use the RGB image features as a search request Q, and gradually search the RGB image features and the depth map features to obtain key value elements; Input the key-value elements into the multi-head attention mechanism module to obtain the confidence of the retrieval results; Based on layer normalization and a multi-layer perceptron, obtaining correlation features between the RGB image features and the depth map features in the confidence; Performing an offset window operation on the associated features and the depth map features to obtain a new retrieval result confidence; Based on the new retrieval result confidence, obtaining the encoded integrated features; The correlation feature between the RGB picture and the depth map is: F _corr =MLP(LN(F _qkv ))+F _qkv Among them, F _corr is the correlation feature, F _qkv is the result of multi-head attention calculation, LN is layer normalization, and MLP is a multi-layer perceptron. 4. The method for capturing transparent objects based on depth completion according to claim 3, wherein performing an offset window operation on the associated features and the depth map features comprises: Move each pixel in the associated features and the depth map features to the lower right corner by two pixels, and then re-block the associated features and the depth map features; The block features are used as new search requests Q, and the index elements K and content elements V corresponding to the block-related features and depth map features are gradually searched to obtain new key-value elements; The new key-value elements are input into the multi-head attention mechanism module to calculate the new retrieval result confidence. 5. The method for grasping transparent objects based on depth completion according to claim 1, wherein the composite distillation loss function is: in, The loss function for the composite distillation is, is the distance loss, is the structural loss, is the difference between the real depth map and the student predicted depth map, is the difference between the predicted depth maps of the teacher and the student, and λ ₁ , λ ₂ , and λ ₃ are the constant weights of the corresponding losses.