CN113099210B - Three-dimensional image restoration method, device, computer equipment and storage medium - Google Patents

Abstract

The present application relates to a three-dimensional image restoration method, device, computer equipment and storage medium. The method includes: acquiring a first three-dimensional image; inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image; and then judging the first three-dimensional image according to the first three-dimensional image and the second three-dimensional image. Whether there is an occluded part or a shadow part in the 3D image; if there is a occluded part or a shadow part in the first 3D image, output a second 3D image. This method can restore the 3D image of the target, and judge whether there is occlusion or shadow in the real 3D image, and then when judging that there is occlusion or shadow in the real 3D image, select and output the restored 3D image according to the judgment result, so as to improve the 3D image The purpose of restoring precision.

Description

Three-dimensional image restoration method, device, computer equipment and storage medium

technical field

The present application relates to the technical field of three-dimensional image processing, in particular to a three-dimensional image restoration method, device, computer equipment and storage medium.

Background technique

With the development of hardware such as depth cameras, laser sensors, and image processors, and the development of deep neural networks, 3D vision technology has been widely used in assisting intelligent driving cars, service robots, and industrial robots. Because these robot systems are in dynamic scenes, these systems often encounter problems such as occlusion, out of view range and light changes during the movement process. Occlusion will cause the loss of target image information, which will make the subsequent recognition, detection and Positioning errors or failures; exceeding the field of view will also cause loss of image information; changes in light will cause errors in target image information.

In the traditional technology, it is common to repair the missing contour of the 3D object model by using the method of Euler arc curve. This method is mainly aimed at the restoration of contour information, and cannot solve the problem of low image restoration accuracy caused by occlusion.

Contents of the invention

Based on this, it is necessary to provide a three-dimensional image restoration method, device, computer equipment and storage medium capable of improving image restoration accuracy in order to address the above technical problems.

A method for restoring a three-dimensional image, the method comprising:

acquiring a first three-dimensional image;

Inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;

According to the first three-dimensional image and the second three-dimensional image, determine whether there is an occluded part or a shadow part in the first three-dimensional image;

If the first three-dimensional image has an occluded part or a shadow part, output the second three-dimensional image.

In one of the embodiments, acquiring the first three-dimensional image includes:

acquire an image of the subject;

Three-dimensional reconstruction is performed on the image of the subject to obtain a first three-dimensional image.

In one of the embodiments, performing three-dimensional reconstruction on the image of the subject to obtain the first three-dimensional image includes:

Obtain the 3D point cloud information of the subject according to the image of the subject;

According to the 3D point cloud information of the shooting object, three-dimensional reconstruction is performed on the image of the shooting object to obtain a first three-dimensional image.

In one of the embodiments, the preset image restoration neural network is obtained by the following method:

Obtain the initial sample image, crop the random area of the initial sample image, and assign black to the cropped area to obtain the sample image;

Input the sample image into the initial image restoration neural network, and process it through the convolutional attention layer, the 3D convolutional network layer and the 3D deconvolutional network layer to obtain the restored sample image;

The network weights of the initial image restoration neural network are adjusted according to the restoration sample image and the initial sample image to obtain a preset image restoration neural network.

In one of the embodiments, according to the first 3D image and the second 3D image, judging whether the first 3D image has an occluded part or a shadow part includes:

Respectively acquire relevant information of the first three-dimensional image and the second three-dimensional image, where the relevant information is color information or histogram information;

Obtaining a related information difference between the first 3D image and the second 3D image according to the related information of the first 3D image and the second 3D image;

According to the relevant information difference between the first 3D image and the second 3D image, it is judged whether there is an occluded part or a shadow part in the first 3D image.

In one of the embodiments, according to the relevant information difference between the first 3D image and the second 3D image, it is judged whether there is an occluded part or a shadow part in the first 3D image, including:

According to the type of the relevant information, the relevant information difference range threshold corresponding to the relevant information is obtained;

If the relevant information difference between the first three-dimensional image and the second three-dimensional image is within the relevant information difference range threshold, it is determined that there is no blocked part or shadow part in the three-dimensional image;

If the relevant information difference between the first 3D image and the second 3D image is not within the relevant information difference range threshold, it is determined that there is an occluded part or a shadow part in the 3D image.

In one embodiment, the method also includes:

If there is no occluded part and no shadow part in the 3D image, output the 3D image.

A three-dimensional image restoration device, the device comprising:

An image acquisition module, configured to acquire the first three-dimensional image;

An image restoration module, configured to input the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;

An image judging module, configured to judge whether there is an occluded part or a shadow part in the first three-dimensional image according to the first three-dimensional image and the second three-dimensional image;

The image output module is configured to output the second three-dimensional image if the first three-dimensional image has an occluded portion or a shadow portion.

A computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

acquiring a first three-dimensional image;

Inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;

According to the first three-dimensional image and the second three-dimensional image, determine whether there is an occluded part or a shadow part in the first three-dimensional image;

If the first three-dimensional image has an occluded part or a shadow part, output the second three-dimensional image.

A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

acquiring a first three-dimensional image;

Inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;

According to the first three-dimensional image and the second three-dimensional image, determine whether there is an occluded part or a shadow part in the first three-dimensional image;

If the first three-dimensional image has an occluded part or a shadow part, output the second three-dimensional image.

The above-mentioned three-dimensional image restoration method, device, computer equipment, and storage medium obtain a first three-dimensional image; input the first three-dimensional image into a preset image restoration neural network for restoration processing, and obtain a second three-dimensional image; and then obtain a second three-dimensional image according to the The first three-dimensional image and the second three-dimensional image, judging whether the first three-dimensional image has an occluded portion or a shadow portion; if the first three-dimensional image has an occluded portion or a shadow portion, outputting the second three-dimensional image . This scheme can restore the 3D image of the target, and judge whether there is occlusion or shadow in the real 3D image, and then select and output the restored 3D image according to the judgment result when judging that there is occlusion or shadow in the real 3D image, so as to improve the restoration of 3D image purpose of precision.

Description of drawings

Fig. 1 is a schematic flow chart of a three-dimensional image restoration method in an embodiment;

Fig. 2 is a schematic flow chart of acquiring a three-dimensional image in one embodiment;

Fig. 3 is a schematic flow chart of establishing a preset image restoration neural network in one embodiment;

Fig. 4 is a schematic structural diagram of a preset image restoration neural network in an embodiment;

Fig. 5 is a schematic flow chart of judging whether there is an occluded part or a shadow part in an embodiment;

Fig. 6 is a schematic flow chart of continuous generation of restored three-dimensional images in an embodiment;

Fig. 7 is a schematic diagram of a scene of three-dimensional image restoration in an industrial environment in an embodiment;

Fig. 8 is a structural block diagram of a three-dimensional image restoration device in an embodiment;

Figure 9 is an internal block diagram of a computer device in one embodiment.

detailed description

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

In one embodiment, as shown in FIG. 1 , a method for restoring a three-dimensional image is provided. In this embodiment, the method is applied to a terminal for illustration. It can be understood that the method can also be applied to a server or It is based on a system including a terminal and a server, and is realized through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

Step 102, acquiring a first three-dimensional image.

Specifically, the processor acquires an image, and establishes a first three-dimensional image on the image, where the first three-dimensional image refers to a real three-dimensional image. In computer vision, creating a 3D image refers to the process of reconstructing 3D information from single-view or multi-view images. The information of a single view is incomplete, so 3D reconstruction from a single view needs to utilize empirical knowledge. The multi-view 3D reconstruction (similar to human binocular positioning) is relatively easy. The method is to calibrate the camera first, that is, calculate the relationship between the camera's image coordinate system and the world coordinate system, and then use multiple 2D images to information to reconstruct three-dimensional information.

Step 104, inputting the first 3D image into a preset image restoration neural network for restoration processing to obtain a second 3D image.

Specifically, the real 3D image is input into a preset image restoration neural network, and the preset image restoration neural network outputs a second 3D image, wherein the second 3D image refers to a restored 3D image.

Among them, the preset image restoration neural network is obtained according to the training set extracted from the three-dimensional image of the sample. The more samples in the training set, the higher the accuracy of the preset image restoration neural network after the training is completed. The preset image restoration The restored 3D image output by the neural network is more accurate. The preset image restoration neural network can be composed of a single neural network, such as a three-dimensional convolutional network; it can also be a composite neural network composed of multiple neural networks, such as multiple three-dimensional convolutional networks and three-dimensional deconvolutional networks composite neural network.

Step 106, according to the first 3D image and the second 3D image, determine whether there is an occluded part or a shadow part in the first 3D image.

Specifically, compare the restored 3D image and the real 3D image, and use an occlusion judgment algorithm to judge whether there is an occluded part or a shadow part in the first real 3D image.

Among them, there are currently many occlusion judgment algorithms, such as the number of iterations judgment method: use the Kalman filter algorithm to estimate its position in the next frame in the current frame image, obtain the estimated position, set the tracking window at the estimated position, and set the tracking window in the tracking window Use the Mean Shift algorithm to search for the position most similar to the missile target template as the precise position of the missile target, judge whether the target is blocked or lost according to the number of iterations of the Mean Shift algorithm, and limit the number of iterations of the algorithm. If the target is not lost, the position is given directly; if the target is not found within the limited number of times, occlusion or loss occurs, and the window width is enlarged to search again. Also for example similarity determination method: for the k frame image, given threshold Th, if Bhattacharyya coefficient ρ (yk) ≥ Th, then there is no occlusion, normal tracking; otherwise if ρ (yk) < Th, it means occlusion occurs; Another example is the residual judgment method: In the current frame, it is judged whether a large proportion of occlusions occurs according to the magnitude of the residual between the estimated value of the Kalman filter on the target position and the measured value of the Kalman filter obtained by the Mean Shift algorithm . In this embodiment, an appropriate occlusion judgment algorithm is selected according to different implementation scenarios.

Step 108, if the first 3D image has an occluded portion or a shadow portion, output a second 3D image.

Specifically, if there is an occluded part or a shadow part in the real 3D image, the real 3D image needs to be restored to generate a 3D image without occlusion and shadow, so the restored 3D image is output. If there is no occluded part or shadow part in the real 3D image, the real 3D image is directly output without restoring the real 3D image.

The above three-dimensional image restoration method obtains the first three-dimensional image; inputs the first three-dimensional image into the preset image restoration neural network for restoration processing, and obtains the second three-dimensional image; then judges the second three-dimensional image according to the first three-dimensional image and the second three-dimensional image. Whether there is a occluded part or a shadow part in a 3D image; if there is a occluded part or a shadow part in the first 3D image, output a second 3D image. This scheme can restore the 3D image of the target, and judge whether there is occlusion or shadow in the real 3D image, and then select and output the restored 3D image according to the judgment result when judging that there is occlusion or shadow in the real 3D image, so as to improve the restoration of 3D image purpose of precision.

In one embodiment, acquiring the first three-dimensional image includes: acquiring an image of the subject; performing three-dimensional reconstruction on the image of the subject to obtain the first three-dimensional image.

Further, performing 3D reconstruction on the image of the subject to obtain the first 3D image includes: acquiring 3D point cloud information of the subject according to the image of the subject; 3D reconstruction to obtain the first 3D image.

In one embodiment, as shown in Figure 2, acquiring the first three-dimensional image includes:

Step 202, acquiring an image of the object to be photographed.

Specifically, an image acquisition device is used to acquire continuous time images of the subject. The image acquisition device may be a camera or a camera. The camera is mounted on the robot system, and the robot system completes multi-target recognition through the information collected by the camera. The robot system can refer to smart driving cars, service robots, industrial robots, etc.; the shooting objects can refer to cars, people, industrial parts, etc. The lighting conditions, the geometric characteristics of the camera, etc. have a great influence on the subsequent image processing.

Step 204, according to the image of the subject, acquire the three-dimensional point cloud information of the subject.

Specifically, three-dimensional point cloud information is also called three-dimensional point cloud data. In some cases, the 3D point cloud information also contains color image information.

Step 206: Perform 3D reconstruction on the image of the subject according to the 3D point cloud information of the subject to obtain a first 3D image.

Specifically, based on the three-dimensional point cloud information of the shooting object at different moments, the three-dimensional reconstruction of the shooting object is performed to obtain the global point cloud information. Then select the local area that overlaps with the global point cloud information for measurement, obtain the local point cloud information and then register and update the global point cloud information, repeat this process until the measurement of all surface areas is completed, and finally the measurement is completed The updated global point cloud data is then globally optimized to obtain a point cloud model.

Optionally, 3D reconstruction is performed according to the image of the subject, and obtaining the reconstructed 3D image usually includes camera calibration, feature extraction, stereo matching and 3D reconstruction.

In one embodiment, as shown in Figure 3, the preset image restoration neural network is obtained by the following method:

Step 302, acquiring an initial sample image, clipping a random area on the initial sample image, and assigning a black value to the cropped area to obtain a sample image.

Specifically, a training set is obtained, and the training set includes at least one sample image for cropping a random region.

Step 304, input the sample image into the initial image restoration neural network, and process through the convolution attention layer, the 3D convolution network layer and the 3D deconvolution network layer to obtain the restored sample image.

Specifically, the image restoration neural network can be composed of a single neural network model, such as a three-dimensional convolutional network; it can also be a composite neural network composed of multiple neural networks, such as multiple three-dimensional convolutional networks and three-dimensional deconvolutional networks composed of composite neural networks.

Step 306 , adjusting the network weights of the initial image restoration neural network according to the restored sample image and the initial sample image to obtain a preset image restoration neural network.

Specifically, the image restoration neural network is trained and acquired according to the training set extracted from sample images.

For example, the image restoration neural network includes sequentially connected input layers, no less than one attention layer, no less than one three-dimensional convolutional network layer, no less than one three-dimensional deconvolution network layer, and an output layer; the attention layer is used for Attention image features are obtained by performing convolutional attention computation on the received image features. The 3D convolutional network layer and the 3D deconvolutional network layer are used for 3D image restoration according to the received image features.

As shown in Figure 4, it is a network topology diagram of the image restoration neural network in one embodiment, in the image restoration neural network in Figure 4, the input layer Input, the convolutional attention layer ConvAttention, the three-dimensional convolutional network layer Conv3D, the volume Product attention layer ConvAttention, 3 3D convolution network layers Conv3D, 2 3D deconvolution network layers Conv3DT, convolution attention layer ConvAttention, 3D deconvolution network layer Conv3DT, convolution attention layer ConvAttention, 2 3D The deconvolution network layer Conv3DT and the output layer Output. Figure 4 contains the basic configuration of the network and the size of the processed tensor (the processed output of each layer in the network becomes a tensor). "2*2*2*64" means that the size of the convolution kernel is 2*2*2, the step size is 2, the number of convolution kernels is 64, and zero padding is required for each layer. For example, "[l/2,w/2,h/2,64]" refers to the dimensions of the processed tensor. The structure of the combination of the 3D convolutional network layer and the 3D deconvolutional network layer is a sequential structure, which can better learn the motion timing relationship of the target 3D image. The introduction of the convolutional attention layer can promote the network to pay more attention to the main part of the 3D image, and improve the ability of the network to learn and restore the 3D image.

For example, suppose an image is input, and its size is [l, w, h, c], where l, w, h, and c refer to the length, width, height, and number of channels, respectively. After connection and ConvAttention processing, the tensor size is [l,w,h,c], and the tensor size is [l/2, w/2,h/2,64], after ConvAttention processing and Conv3D processing of 128 convolution kernels, the tensor size is [l/4,w/4,h/4,128], after 256 2*2* 2 After Conv3D processing with a convolution kernel step size of 2, the tensor size is [l/8, w/8, h/8, 256], after 512 Conv3D processing with a 2*2*2 convolution kernel step size of 2 The obtained tensor size is [l/16,w/16,h/16,512], after 1024 Conv3DT processing with 2*2*2 convolution kernel step size 2, the tensor size is [l/8,w/ 8, h/8, 1024], after 512 Conv3DT processing with 2*2*2 convolution kernel step size 2, the tensor size is [l/4, w/4, h/4, 512], after ConvAttention processing After Conv3DT processing with 256 convolution kernels, the tensor size is [l/2, w/2, h/2, 256], after ConvAttention processing and Conv3DT processing with 128 convolution kernels, the tensor size is [l, w,h,128], after c Conv3DT processing with a 2*2*2 convolution kernel step size of 2, the tensor size is [l,w,h,c], and the tensor is output.

In one embodiment, according to the first 3D image and the second 3D image, judging whether there is an occluded part or a shadow part in the first 3D image includes: obtaining relevant information of the first 3D image and the second 3D image respectively, and the relevant information is color information or histogram information; according to the relevant information of the first 3D image and the second 3D image, the relevant information difference between the first 3D image and the second 3D image is obtained; according to the relevant information of the first 3D image and the second 3D image Poor, determine whether there is an occluded part or a shadow part in the first 3D image.

Further, according to the relevant information difference between the first 3D image and the second 3D image, judging whether there is an occluded part or a shadow part in the first 3D image, including: according to the type of relevant information, obtaining the relevant information difference range corresponding to the relevant information Threshold; if the relevant information difference between the first three-dimensional image and the second three-dimensional image is in the relevant information difference range threshold, it is determined that there is no occluded part or shadow part in the three-dimensional image; if the relevant information difference between the first three-dimensional image and the second three-dimensional image If it is not within the relevant information difference range threshold, it is determined that there is an occluded part or a shadow part in the three-dimensional image.

In one embodiment, as shown in FIG. 5, according to the first three-dimensional image and the second three-dimensional image, judging whether there is an occluded part or a shadow part in the first three-dimensional image includes:

In step 502, relevant information of the first 3D image and the second 3D image are obtained respectively, and the relevant information is color information or histogram information or other information.

Specifically, color information or histogram information of the real 3D image and the restored 3D image are acquired respectively.

Step 504, according to the relevant information of the first 3D image and the second 3D image, obtain the difference of relevant information between the first 3D image and the second 3D image.

Specifically, according to the color information or histogram information of the real 3D image and the restored 3D image, a color information difference or a histogram information difference between the real 3D image and the restored 3D image is obtained.

Step 506, according to the type of the relevant information, obtain the relevant information difference range threshold corresponding to the relevant information.

Specifically, according to the characteristics of color information and histogram information, set different relevant information difference thresholds. If the current acquisition is color information, set a color information difference threshold ±M%. If the current acquisition is histogram information , set a histogram information difference threshold ±N%, usually, M and N are not less than 3.

Step 508, if the relevant information difference between the first 3D image and the second 3D image is within the relevant information difference range threshold, it is determined that there is no occluded part or shadow part in the 3D image; if the relevant information of the first 3D image and the second 3D image If the difference is not within the relevant information difference range threshold, it is determined that there is an occluded part or a shadow part in the 3D image.

Specifically, for example, if the color information difference between the real 3D image and the restored 3D image is within the color information difference threshold range, it is determined that there is no occluded or shadowed portion in the real 3D image; if the color information difference between the real 3D image and the restored 3D image is If it is not within the color information difference threshold range, it is determined that there is an occluded part or a shadow part in the real 3D image.

In one embodiment, the method further includes: if there is no occluded part and no shadow part in the three-dimensional image, outputting the three-dimensional image.

Specifically, if there is no occluded part and no shadow part in the real 3D image, the original image does not need to be restored, so the real 3D image is directly output.

In one embodiment, as shown in FIG. 6 , a method for restoring a three-dimensional image is provided. The method is illustrated by being applied to an industrial environment as shown in FIG. 7 . The method includes: acquiring an image of a workpiece on a conveyor belt. Perform 3D reconstruction on the image of the workpiece to obtain a real 3D image. Obtain a real 3D image, input the real 3D image into the preset image restoration neural network for restoration processing, and obtain a restored 3D image; according to the real 3D image and the restored 3D image, judge whether there is an occluded part or a shadow part in the real 3D image; if true If there are occluded or shadowed parts in the 3D image, the restored 3D image will be output.

Specifically, the conveyor belt starts to run, and the automatic device starts to place workpieces on the conveyor belt. Start the restoration of the 3D image of the workpiece. The 3D camera on the industrial robot collects the 3D point cloud of the workpiece on the conveyor belt, and performs 3D reconstruction on the 3D point cloud to obtain a 3D image i. Use the 3D image i to input the preset image restoration neural network to obtain the restored 3D image i', compare i and i' according to the occlusion judgment algorithm to judge whether the state of the current workpiece is in the occlusion state, and compare the color information of i and i', if If the color information difference between the two is within the range of ±M% (M≥3), it is considered that the workpiece is not in a blocking state, and i is output. If the color information difference ratio of the two is in the range of ±M% (M≥3), it is considered that the workpiece is in a blocking state, and i' is output. Determine whether to continue image acquisition, if necessary, acquire the next image, and perform the above restoration process on the next image, if not, stop the restoration of the three-dimensional image of the workpiece.

In this embodiment, by acquiring a real 3D image; inputting the real 3D image into a preset image restoration neural network for restoration processing, a restored 3D image is obtained; and then judging whether the real 3D image is blocked according to the real 3D image and the restored 3D image part or shaded part; if the real 3D image has an occluded part or a shadowed part, output the restored 3D image. This scheme can restore the 3D image of the target, and judge whether there is occlusion or shadow in the real 3D image, and then select and output the restored 3D image according to the judgment result when judging that there is occlusion or shadow in the real 3D image, so as to improve the restoration of 3D image purpose of precision.

It should be understood that although the various steps in the flow charts of FIGS. 1-3 and 5-6 are displayed sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Figures 1-3, 5-6 may include multiple steps or stages, these steps or stages are not necessarily executed at the same time, but may be executed at different times, these steps Or the execution sequence of the stages is not necessarily performed sequentially, but may be executed in turn or alternately with other steps or at least a part of steps or stages in other steps.

In one embodiment, as shown in FIG. 8 , a three-dimensional image restoration device 800 is provided, including: an image acquisition module 801, an image restoration module 802, an image judgment module 803, and an image output module 804,

in:

An image acquiring module 801, configured to acquire a first three-dimensional image.

The image restoration module 802 is configured to input the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image.

The image judging module 803 is configured to judge whether there is an occluded part or a shadow part in the first three-dimensional image according to the first three-dimensional image and the second three-dimensional image.

An image output module 804, configured to output a second 3D image if there is an occluded portion or a shadow portion in the first 3D image.

In one embodiment, the image acquisition module 801 includes:

The image capture sub-module is used to acquire images of objects to be photographed.

The three-dimensional reconstruction sub-module is used to perform three-dimensional reconstruction on the image of the subject to obtain the first three-dimensional image.

In one embodiment, the three-dimensional reconstruction submodule also includes:

The three-dimensional information acquisition unit is configured to acquire the three-dimensional point cloud information of the object according to the image of the object.

The three-dimensional image reconstruction unit is configured to perform three-dimensional reconstruction on the image of the object to obtain the first three-dimensional image according to the three-dimensional point cloud information of the object.

In one embodiment, the image restoration module 802 includes:

The sample acquisition sub-module is used to acquire an initial sample image, crop a random area of the initial sample image, and assign a black color to the cropped area to obtain a sample image.

The model training sub-module is used to input the sample image into the initial image restoration neural network, and obtain the restored sample image through the convolution attention layer, the 3D convolution network layer and the 3D deconvolution network layer.

The model generation sub-module is used to adjust the network weights of the initial image restoration neural network according to the restored sample image and the initial sample image to obtain a preset image restoration neural network.

In one embodiment, the image judging module 803 includes:

The related information acquiring sub-module is used to acquire related information of the first three-dimensional image and the second three-dimensional image respectively, and the related information is color information or histogram information.

The information difference obtaining sub-module is used to obtain the relevant information difference between the first 3D image and the second 3D image according to the relevant information of the first 3D image and the second 3D image.

The information difference judging sub-module is used to judge whether there is an occluded part or a shadow part in the first 3D image according to the related information difference between the first 3D image and the second 3D image.

In one embodiment, the information difference determination submodule also includes:

The range threshold setting unit is configured to acquire the related information difference range threshold corresponding to the related information according to the type of the related information.

A range threshold judging unit, configured to determine that there is no occluded part or shadow part in the three-dimensional image if the relevant information difference between the first three-dimensional image and the second three-dimensional image is within the relevant information difference range threshold; if the first three-dimensional image and the second three-dimensional If the relevant information difference of the image is not within the relevant information difference range threshold, it is determined that there is an occluded part or a shadow part in the three-dimensional image.

In one embodiment, the image output module 804 is further configured to output the three-dimensional image if there is no occluded part and no shadow part in the three-dimensional image.

For specific limitations on the three-dimensional image restoration apparatus, refer to the above-mentioned limitations on the three-dimensional image restoration method, which will not be repeated here. Each module in the above-mentioned three-dimensional image restoration device may be fully or partially realized by software, hardware or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure may be as shown in FIG. 9 . The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, an operator network, NFC (Near Field Communication) or other technologies. When the computer program is executed by a processor, a method for restoring a three-dimensional image is realized. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer device , and can also be an external keyboard, touchpad or mouse.

Those skilled in the art can understand that the structure shown in FIG. 9 is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer equipment on which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, a computer program is stored in the memory, and the processor implements the following steps when executing the computer program:

acquiring a first three-dimensional image;

Inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;

According to the first three-dimensional image and the second three-dimensional image, determine whether there is an occluded part or a shadow part in the first three-dimensional image;

If the first three-dimensional image has an occluded part or a shadow part, output the second three-dimensional image.

In one embodiment, the following steps are also implemented when the processor executes the computer program:

acquire an image of the subject;

Three-dimensional reconstruction is performed on the image of the subject to obtain a first three-dimensional image.

In one embodiment, the following steps are also implemented when the processor executes the computer program:

Obtain the 3D point cloud information of the subject according to the image of the subject;

According to the 3D point cloud information of the shooting object, three-dimensional reconstruction is performed on the image of the shooting object to obtain a first three-dimensional image.

In one embodiment, the following steps are also implemented when the processor executes the computer program:

Obtain the initial sample image, crop the random area of the initial sample image, and assign black to the cropped area to obtain the sample image;

Input the sample image into the initial image restoration neural network, and process it through the convolutional attention layer, the 3D convolutional network layer and the 3D deconvolutional network layer to obtain the restored sample image;

The network weights of the initial image restoration neural network are adjusted according to the restoration sample image and the initial sample image to obtain a preset image restoration neural network.

In one embodiment, the following steps are also implemented when the processor executes the computer program:

Respectively acquire relevant information of the first three-dimensional image and the second three-dimensional image, where the relevant information is color information or histogram information;

Obtaining a related information difference between the first 3D image and the second 3D image according to the related information of the first 3D image and the second 3D image;

According to the relevant information difference between the first 3D image and the second 3D image, it is judged whether there is an occluded part or a shadow part in the first 3D image.

In one embodiment, the following steps are also implemented when the processor executes the computer program:

According to the type of the relevant information, the relevant information difference range threshold corresponding to the relevant information is obtained;

If the relevant information difference between the first three-dimensional image and the second three-dimensional image is within the relevant information difference range threshold, it is determined that there is no blocked part or shadow part in the three-dimensional image;

If the relevant information difference between the first 3D image and the second 3D image is not within the relevant information difference range threshold, it is determined that there is an occluded part or a shadow part in the 3D image.

In one embodiment, the following steps are also implemented when the processor executes the computer program:

If there is no occluded part and no shadow part in the 3D image, output the 3D image.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

acquiring a first three-dimensional image;

Inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;

According to the first three-dimensional image and the second three-dimensional image, determine whether there is an occluded part or a shadow part in the first three-dimensional image;

If the first three-dimensional image has an occluded part or a shadow part, output the second three-dimensional image.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented:

acquire an image of the subject;

Three-dimensional reconstruction is performed on the image of the subject to obtain a first three-dimensional image.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented:

Obtain the 3D point cloud information of the subject according to the image of the subject;

According to the 3D point cloud information of the shooting object, three-dimensional reconstruction is performed on the image of the shooting object to obtain a first three-dimensional image.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented:

Obtain the initial sample image, crop the random area of the initial sample image, and assign black to the cropped area to obtain the sample image;

Input the sample image into the initial image restoration neural network, and process it through the convolutional attention layer, the 3D convolutional network layer and the 3D deconvolutional network layer to obtain the restored sample image;

The network weights of the initial image restoration neural network are adjusted according to the restoration sample image and the initial sample image to obtain a preset image restoration neural network.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented:

Respectively acquire relevant information of the first three-dimensional image and the second three-dimensional image, where the relevant information is color information or histogram information;

Obtaining a related information difference between the first 3D image and the second 3D image according to the related information of the first 3D image and the second 3D image;

According to the relevant information difference between the first 3D image and the second 3D image, it is judged whether there is an occluded part or a shadow part in the first 3D image.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented:

According to the type of the relevant information, the relevant information difference range threshold corresponding to the relevant information is obtained;

If the relevant information difference between the first three-dimensional image and the second three-dimensional image is within the relevant information difference range threshold, it is determined that there is no blocked part or shadow part in the three-dimensional image;

If the relevant information difference between the first 3D image and the second 3D image is not within the relevant information difference range threshold, it is determined that there is an occluded part or a shadow part in the 3D image.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented:

If there is no occluded part and no shadow part in the 3D image, output the 3D image.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include at least one of non-volatile memory and volatile memory. The non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, and the like. Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered to be within the range described in this specification.

The above-mentioned embodiments only represent several implementation modes of the present application, and the description thereof is relatively specific and detailed, but it should not be construed as limiting the scope of the patent for the invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of protection of the patent application should be based on the appended claims.

Claims (10)

1. A method for three-dimensional image restoration, the method comprising:

acquiring a first three-dimensional image;

inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;

respectively acquiring related information of the first three-dimensional image and the second three-dimensional image, wherein the related information is color information or histogram information; obtaining a relevant information difference between the first three-dimensional image and the second three-dimensional image according to the relevant information of the first three-dimensional image and the second three-dimensional image; judging whether the first three-dimensional image has an occluded part or a shadow part according to the related information difference between the first three-dimensional image and the second three-dimensional image;

and if the first three-dimensional image has an occluded part or a shadow part, outputting the second three-dimensional image.

2. The method of claim 1, wherein said acquiring a first three-dimensional image comprises:

acquiring an image of a photographic subject;

and performing three-dimensional reconstruction on the image of the shooting object to obtain a first three-dimensional image.

3. The method of claim 2, wherein the three-dimensional reconstruction of the image of the photographic subject to obtain a first three-dimensional image comprises:

acquiring three-dimensional point cloud information of the shot object according to the image of the shot object;

and performing three-dimensional reconstruction on the image of the shot object according to the three-dimensional point cloud information of the shot object to obtain a first three-dimensional image.

4. The method of claim 1, wherein the predetermined image restoration neural network is obtained by:

obtaining an initial sample image, cutting a random area of the initial sample image, and assigning the cut area to be black to obtain a sample image;

inputting the sample image into an initial image restoration neural network, and processing the sample image through a convolution attention layer, a three-dimensional convolution network layer and a three-dimensional deconvolution network layer to obtain a restored sample image;

and adjusting the network weight of the initial image reduction neural network according to the reduction sample image and the initial sample image to obtain a preset image reduction neural network.

5. The method according to claim 1, wherein the determining whether the first three-dimensional image has the blocked part or the shadow part according to the difference between the related information of the first three-dimensional image and the second three-dimensional image comprises:

acquiring a related information difference range threshold corresponding to the related information according to the type of the related information;

if the relevant information difference between the first three-dimensional image and the second three-dimensional image is within the relevant information difference range threshold, judging that an occluded part or a shadow part does not exist in the three-dimensional image;

and if the relevant information difference between the first three-dimensional image and the second three-dimensional image is not within the relevant information difference range threshold, judging that an occluded part or a shadow part exists in the three-dimensional image.

6. The method of claim 1, further comprising:

and if the first three-dimensional image does not have the blocked part and the shadow part, outputting the first three-dimensional image.

7. A three-dimensional image restoration device, the device comprising:

the image acquisition module is used for acquiring a first three-dimensional image;

the image restoration module is used for inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;

the image judging module is used for respectively acquiring relevant information of the first three-dimensional image and the second three-dimensional image, wherein the relevant information is color information or histogram information; obtaining a relevant information difference between the first three-dimensional image and the second three-dimensional image according to the relevant information of the first three-dimensional image and the second three-dimensional image; judging whether the first three-dimensional image has an occluded part or a shadow part according to the related information difference between the first three-dimensional image and the second three-dimensional image;

and the image output module is used for outputting the second three-dimensional image if the first three-dimensional image has a blocked part or a shadow part.

8. The apparatus according to claim 7, wherein the image determining module is further configured to obtain a related information difference range threshold corresponding to the related information according to a type of the related information; if the relevant information difference between the first three-dimensional image and the second three-dimensional image is within the relevant information difference range threshold, judging that an occluded part or a shadow part does not exist in the three-dimensional image; and if the relevant information difference between the first three-dimensional image and the second three-dimensional image is not within the relevant information difference range threshold, judging that an occluded part or a shadow part exists in the three-dimensional image.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 6.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.

Images