CN110335211B - Depth image correction method, terminal device and computer storage medium - Google Patents

Abstract

The embodiment of the present application discloses a depth image correction method, a terminal device, and a computer storage medium. The method includes: acquiring an original image corresponding to a target object and a main color image and a sub-color image corresponding to the target object; wherein the original image is According to the acquisition of the target object by the TOF sensor, the main color image and the sub-color image are obtained from the acquisition of the target object by the dual cameras; according to the main color image and the sub-color image, the preset dual-camera algorithm is used to determine the target object. Corresponding first depth information and first confidence; according to the original image, determine the second depth information corresponding to the target object; based on the first depth information, the second depth information and the first confidence, determine the error in the first depth information data area; correcting the wrong data area by using the second depth information and the main color image to obtain target depth information, and obtaining a depth image according to the target depth information.

Description

Depth image correction method, terminal device and computer storage medium

technical field

The present application relates to the technical field of image processing, and in particular, to a depth image correction method, a terminal device, and a computer storage medium.

Background technique

With the rapid development of smart terminals, terminal devices such as mobile phones, handheld computers, digital cameras, and video cameras have become an indispensable tool in users' lives, and have brought great convenience to all aspects of users' lives. Existing terminal devices basically have a photographing function, and users can use the terminal device to shoot various images.

When shooting an image with a blur effect, the terminal device is usually required to be equipped with dual cameras. Obtaining depth information through dual cameras, although the structure is simple, the hardware power consumption is low, and the resolution is high, there are still defects, such as poor adaptability to scenes without texture, repeated texture, overexposure, underexposure, etc., resulting in The accuracy of the acquired depth information is low, which affects the effect of portrait blurring.

SUMMARY OF THE INVENTION

The main purpose of this application is to propose a depth image correction method, a terminal device and a computer storage medium, which can repair the phenomenon of depth errors in areas such as no texture, repeated texture, overexposure, underexposure, etc. Thereby, the accuracy of depth in dual-camera portrait mode is improved, and the accuracy of portrait blurring is further optimized.

In order to achieve the above-mentioned purpose, the technical scheme of the present application is achieved in this way:

In a first aspect, an embodiment of the present application provides a method for correcting a depth image, the method comprising:

Obtain the original image corresponding to the target object and the main color image and the sub-color image corresponding to the target object; wherein, the original image is obtained according to the acquisition of the target object by the TOF sensor, and the main color image and the sub-color image are obtained. The image is obtained according to the acquisition of the target object by the dual cameras;

According to the primary color image and the secondary color image, the preset dual-camera algorithm is used to determine the first depth information and the first confidence level corresponding to the target object; according to the original image, the first depth information corresponding to the target object is determined. 2. depth information;

determining an erroneous data region in the first depth information based on the first depth information, the second depth information and the first confidence level;

Correction processing is performed on the erroneous data area by using the second depth information and the main color image to obtain target depth information, and a depth image is obtained according to the target depth information.

In a second aspect, an embodiment of the present application provides a terminal device, where the terminal device includes: an acquisition unit, a determination unit, and a correction unit, wherein,

The acquisition unit is configured to acquire the original image corresponding to the target object and the main color image and the sub-color image corresponding to the target object; wherein, the original image is obtained according to the acquisition of the target object by the TOF sensor, and the main color image and the sub-color image corresponding to the target object are obtained; The color image and the sub-color image are obtained according to the acquisition of the target object by the dual cameras;

The determining unit is configured to use a preset dual-camera algorithm to determine the first depth information and the first confidence level corresponding to the target object according to the primary color image and the secondary color image; second depth information corresponding to the target object; and further configured to determine an erroneous data region in the first depth information based on the first depth information, the second depth information and the first confidence level;

The correction unit is configured to perform correction processing on the erroneous data area by using the second depth information and the main color image to obtain target depth information, and obtain a depth image according to the target depth information.

In a third aspect, an embodiment of the present application provides a terminal device, where the terminal device includes: a memory and a processor; wherein,

the memory for storing a computer program executable on the processor;

The processor is configured to execute the depth image correction method according to the first aspect when running the computer program.

In a fourth aspect, an embodiment of the present application provides a computer storage medium, where the computer storage medium stores a depth image correction program, and the depth image correction program is implemented as described in the first aspect when the depth image correction program is executed by at least one processor The correction method of the depth image.

A method for correcting a depth image, a terminal device, and a computer storage medium provided by the embodiments of the present application obtain an original image corresponding to a target object and a main color image and a sub-color image corresponding to the target object; wherein, the original image The image is obtained according to the acquisition of the target object by the TOF sensor, and the main color image and the sub-color image are obtained according to the acquisition of the target object by the dual cameras; Suppose a dual-camera algorithm determines the first depth information and the first confidence level corresponding to the target object; according to the original image, determine the second depth information corresponding to the target object; and then based on the first depth information, the The second depth information and the first confidence level are used to determine the erroneous data area in the first depth information; finally, the erroneous data area is corrected through the second depth information and the main color image to obtain Target depth information, and obtain a depth image according to the target depth information; in this way, by optimizing the first depth information through the second depth information, the depth information in the dual-camera portrait mode can be repaired without texture, repeated texture, overexposure, and underexposure. This improves the accuracy of depth in dual-camera portrait mode; in addition, the target depth information is mainly used for the blurring of the main color image, and it can also optimize the accuracy of portrait blurring and improve the Portrait blur effect.

Description of drawings

FIG. 1 is a schematic view of a blasting structure of a TOF camera provided by an embodiment of the application;

FIG. 2 is a schematic structural diagram of a dual-camera blurring process provided by an embodiment of the present application;

3 is a schematic flowchart of a method for correcting a depth image according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a hardware structure of a terminal device according to an embodiment of the present application;

5 is a schematic diagram of the comparison of the effects of a kind of polar line correction provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of the effect of a dual-camera parallax calculation provided by an embodiment of the present application;

7 is a schematic diagram of a model for calculating depth information according to an embodiment of the present application;

FIG. 8 is a detailed schematic flowchart of a method for correcting a depth image provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of a comparison of the effect of blurring a portrait provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of the composition and structure of a terminal device provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of a hardware structure of another terminal device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

In recent years, due to the rapid development of Time of Flight (TOF) technology, people's research on optics has become more and more in-depth. As a three-dimensional (Three Dimension, 3D) imaging technology, TOF is widely used in terminal devices such as smart phones, PDAs, tablet computers, digital cameras, etc., which can realize distance measurement, 3D modeling, photo blur and somatosensory games, etc. In the application, related applications of AR glasses can also be implemented with augmented reality (Augmented Reality, AR) technology.

Generally speaking, a TOF camera can be composed of a light emitting module and a light receiving module. Among them, the light emitting module is also called a laser transmitter, a TOF transmitter or an emission lighting module, etc., and the light receiving module is also called a detector, a TOF receiver or a photosensitive receiving module, and the like. Specifically, the light emission module emits modulated near-infrared light, which is reflected after encountering the object to be photographed, and then the light receiving module calculates the time difference or phase difference between light emission and reflection, and then converts it to obtain the distance of the object to be photographed. , resulting in depth information.

Referring to FIG. 1 , it shows a schematic diagram of an explosive structure of a TOF camera provided by an embodiment of the present application. As shown in FIG. 1 , the TOF camera 10 includes a light emitting module 110 and a light-emitting receiving module 120; wherein, the light emitting module 110 is composed of a diffuser, a photo-diode (PD), a vertical cavity surface emitting laser (Vertical Cavity Surface Emitting Laser, VCSEL) and a ceramic package, etc.; the light receiving module 120 is composed of a lens, a 940nm narrow-band filter, a TOF sensor, and the like. Those skilled in the art can understand that the composition shown in FIG. 1 does not constitute a limitation on the TOF camera, and the TOF camera may include more or less components than the one shown, or combine some components, or arrange different components .

It can be understood that TOF can be divided into direct time-of-flight (Direct-TOF, D-TOF) and indirect time-of-flight (Indirect-TOF, I-TOF) according to different obtained signal results. Among them, D-TOF obtains the time difference, and I-TOF obtains the phase offset of the return signals of different targets (for example, the proportion of charge or voltage in different phases), so as to calculate the distance of the photographed object and generate depth information. .

通过该相位差再换算得到被拍摄物体的距离D；其中，In addition, according to different modulation methods, I-TOF can be further divided into a pulsed modulation (Pulsed Modulation) scheme and a continuous wave modulation (Continuous Wave Modulation) scheme. At present, the mainstream method used by the terminal equipment of most manufacturers is the indirect TOF scheme of continuous wave modulation (which can be represented by CW-I-TOF). For the CW-I-TOF scheme, each pixel contains 2 capacitors, and the light-emitting module emits 4-segment square wave pulses with a pulse period of Δt; while the light-receiving module has a phase delay when receiving the pulses, each The window phase is delayed by 90°, that is, a quarter of the pulse period (represented by Δt/4), so that the phase delays are 0°, 180°, 90° and 270°, also known as the four-phase method. During exposure, the two capacitors of each pixel are charged alternately, the exposure time is equal, and the difference between the exposure amounts of the two capacitors is recorded as Q1, Q2, Q3, and Q4, respectively; using the relationship between the charge difference and the flight phase, the phase can be calculated Difference

The distance D of the object to be photographed is obtained by converting the phase difference; wherein,

和

表示，将

扩展为

将

扩展为

那么将会存在一个真实的距离以使得两者对应的距离差最小，从而可以确定出真实的距离。When the angle corresponding to the distance of the object to be photographed exceeds 2π, two phases with different frequencies are required to solve the real distance. It is assumed that the two phase values obtained are respectively

and

said, will

expands to

Will

expands to

Then there will be a real distance to minimize the corresponding distance difference between the two, so that the real distance can be determined.

As an active depth sensor, TOF has been widely used in terminal devices such as mobile phones. For example, a manufacturer uses it as a solution for a rear depth sensor. However, the TOF sensor has a low resolution and cannot directly adapt to applications that require high accuracy of foreground edges, such as blurring and matting. Therefore, the dual-camera solution is still the main solution.

The dual-camera solution can include a main camera and a sub-camera. Both the main camera and the sub-camera can be RGB cameras, where RGB represents three channels of red (Red, R), green (Green, G), and blue (Blue, B). The colors of these three channels are mixed or superimposed in different proportions, and all the colors perceived by human vision in the image can be obtained. As a portrait blur application, the dual-camera solution has become a standard function of terminal equipment. The dual-camera blur process is shown in Figure 2. Through the four steps of dual-camera calibration, polar line correction, stereo matching and bokeh blur, This completes the function of double-camera blurring. However, the acquisition of depth information through the dual-camera scheme has many advantages, such as simple structure, low hardware power consumption, high resolution of depth information, and can also take into account most indoor and outdoor scenes; The adaptability of scenes such as repeated texture, overexposure, and underexposure is poor, so that the depth information generated by the dual-camera scheme in areas without texture, repeated texture, overexposure, and underexposure may have errors, resulting in blurring errors.

An embodiment of the present application provides a depth image correction method, and the method is applied to a terminal device. By acquiring the original image corresponding to the target object and the main color image and sub-color image corresponding to the target object; wherein, the original image is obtained according to the acquisition of the target object by the TOF sensor, the main color image and the sub-color image is obtained according to the acquisition of the target object by the dual cameras; then, according to the main color image and the secondary color image, the preset dual-camera algorithm is used to determine the first depth information and the first confidence level corresponding to the target object; according to For the original image, determine the second depth information corresponding to the target object; and then determine the error in the first depth information based on the first depth information, the second depth information and the first confidence level data area; finally, the error data area is corrected through the second depth information and the main color image to obtain target depth information, and a depth image is obtained according to the target depth information; in this way, through the second depth information The first depth information is optimized, which can fix the phenomenon of depth error in areas without texture, repeated texture, overexposure and underexposure in the dual-camera portrait mode, thereby improving the depth accuracy in the dual-camera portrait mode; in addition, The target depth information is mainly used for the blurring processing of the main color image, and can also optimize the accuracy of portrait blurring and improve the effect of portrait blurring.

The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Referring to FIG. 3 , it shows a schematic flowchart of a depth image correction method provided by an embodiment of the present application. As shown in Figure 3, the method may include:

S301: Acquire an original image corresponding to a target object and a main color image and a sub-color image corresponding to the target object; wherein, the original image is obtained according to the acquisition of the target object by a TOF sensor, and the main color image and the sub-color The image is obtained according to the acquisition of the target object by the dual cameras;

It should be noted that this method is applied to a terminal device, and the terminal device includes components such as a TOF sensor and dual cameras (a main camera and a sub-camera). In this way, the original image corresponding to the target object can be acquired by the TOF sensor, and the main color image and the sub-color image corresponding to the target object can be acquired by dual cameras, which is convenient for subsequent calculation of depth information.

It should also be noted that the terminal device may be implemented in various forms. For example, the terminal devices described in this application may include mobile terminals such as mobile phones, tablet computers, notebook computers, PDAs, personal digital assistants (PDAs), wearable devices, digital cameras, video cameras, etc., as well as mobile terminals such as digital TVs , desktop computers and other fixed terminals; the embodiments of this application are not specifically limited.

In some embodiments, for S301, the obtaining the original image corresponding to the target object and the primary color image and the secondary color image corresponding to the target object may include:

S301a: collect the target object through the TOF sensor, and obtain the original image corresponding to the target object;

S301b: Collect the target object through dual cameras, and obtain a main color image corresponding to the target object under the main camera and a sub-color image corresponding to the target object under the sub camera; wherein the dual cameras include a main color image. camera and sub-camera.

It should be noted that, through the acquisition of the target object by the TOF sensor, the original image corresponding to the target object can be obtained, such as a set of RAW images; through the acquisition of the target object by the dual cameras, the corresponding main color image under the main camera can be obtained, such as An RGB main image; and the corresponding sub-color image under the sub-camera, such as an RGB sub-image. In this way, the depth information corresponding to the dual-camera mode can be obtained according to the main color image and the sub-color image collected by the dual cameras, which is represented by the first depth information in the embodiment of the present application; the depth information corresponding to the TOF mode can be obtained according to the original image collected by the TOF sensor , which is represented by the second depth information in this embodiment of the present application.

4, which shows a schematic diagram of a hardware structure of a terminal device provided by an embodiment of the present application. As shown in FIG. 4 , the terminal device may include an application processor (Application Processor, AP), a main camera, a sub-camera, a TOF sensor, and a laser (Laser) transmitter; wherein, the AP side includes a first image signal processor ( First Image Signal Processor (ISP1), Second Image Signal Processor (ISP2), and Mobile Industry Processor Interface (MIPI); in addition, preset algorithms are placed on the AP side, such as The preset dual-camera algorithm, the preset calibration algorithm, etc., are not specifically limited in the embodiments of the present application.

Combined with the terminal device shown in Figure 4, in the dual-camera mode, when the terminal device captures images of the target object, the AP side can connect two cameras respectively through two ISPs to obtain two channels of RGB data, while ensuring dual cameras. Frame synchronization and 3A synchronization in the shooting mode; wherein, 3A synchronization includes automatic focus (Automatic Focus, AF), automatic exposure (Automatic Exposure, AE) and automatic white balance (Automatic White Balance, AWB) synchronization. In Figure 4, the main camera collects the target object, and sends the obtained main color image (one channel of RGB data) to ISP1; the sub-camera collects the target object, and the obtained sub-color image (other All the way RGB data) into ISP2. In addition, the terminal device can also drive the integrated circuit (IC) to ensure the exposure timing requirements of the laser (Laser) and the infrared (Infrared Radiation, IR), and require the IR exposure to be synchronized with the RGB exposure corresponding to the main camera; specifically , which can be implemented by software synchronization or hardware synchronization. Combined with the preset algorithm placed on the AP side, the first depth information corresponding to the target object can be obtained by calculation. In the TOF mode, the terminal device can acquire a set of RAW images through the TOF sensor, so as to obtain the second depth information corresponding to the target object. In this way, in the future, according to the depth fusion of the first depth information and the second depth information in the main camera coordinate system, the error area in the first depth information in the dual camera mode can be corrected, so as to achieve the purpose of improving the depth accuracy.

S302: According to the primary color image and the secondary color image, use a preset dual-camera algorithm to determine first depth information and a first confidence level corresponding to the target object; according to the original image, determine that the target object corresponds to the second depth information of ;

It should be noted that the first confidence level is used to represent the accuracy of the first depth information, and the preset dual-camera algorithm is used to represent a preset algorithm or model based on dual-camera stereo matching. Specifically, according to the primary color image and the secondary color image, the first depth information and the first confidence level are calculated by a preset dual-camera algorithm, and the first depth information and the first confidence level in the dual-camera mode can be obtained; and By calculating the depth information according to the original image, the second depth information in the TOF mode can be obtained; in this way, the first depth information in the dual-camera mode can be optimized subsequently according to the second depth information in the TOF mode.

It should also be noted that the first depth information is calculated based on the main color image and the sub-color image, and the obtained first depth information is originally in the main camera coordinate system, so there is no need to coordinate the first depth. The second depth information is calculated based on the original image, and the obtained second depth information is in the TOF coordinate system, so it is necessary to perform coordinate system transformation on the second depth to align it to the main in the camera coordinate system.

Generally speaking, the higher the resolution of the main camera, the higher the resolution of the depth information it generates. Taking a 4-megapixel camera as an example, its resolution is 2584×1938; while the TOF resolution is lower, and the depth information it generates has a higher resolution. The resolution is also low, for example, the resolution is 320*240; that is, the resolution of the first depth information is higher than the resolution of the second depth information. In this way, after the second depth information is aligned in the main camera coordinate system, the pixels corresponding to the second depth information are sparse, which can also provide some sparse effective pixels for the follow-up.

S303: Determine an erroneous data region in the first depth information based on the first depth information, the second depth information, and the first confidence level;

It should be noted that the erroneous data area refers to an area composed of pixels with depth errors in the first depth information, and the erroneous data area is usually in the low confidence area of the first depth information; wherein, the low confidence area is determined by the first confidence level.

Because in the dual-camera mode, the first depth information has depth errors in areas without texture, repeated textures, etc. At this time, in order to improve the accuracy of the depth in the dual-camera mode, the embodiment of the present application needs to determine the erroneous data area, It is convenient to perform depth calibration on the erroneous data area subsequently. In this way, after both the first depth information and the second depth information are aligned in the main camera coordinate system, a low-confidence region in the first depth information can be determined according to the first confidence; then, in the low-confidence region, Using the first depth information and the second depth information, the erroneous data area in the first depth information can be calculated, which facilitates subsequent calibration of the erroneous data area.

S304: Perform correction processing on the erroneous data area by using the second depth information and the main color image to obtain target depth information, and obtain a depth image according to the target depth information.

It should be noted that, after the erroneous data region in the first depth information is obtained, interpolation and repair processing can be performed on the pixels in the erroneous data region through the second depth information to obtain new depth information. Wherein, the erroneous data area in the first depth information is obtained by interpolation and repair processing of the second depth information, while the non-error data area in the first depth information retains the original first depth information, so that for For the first depth information, new depth information is obtained; it can be seen that the new depth information is obtained by the fusion of the first depth information and the second depth information; in order to reduce the traces of artificial synthesis, the main color image can also be used. The new depth information is filtered, and the final output depth information is the target depth information, and the required depth image is obtained according to the target depth information, thus solving the problem of no texture, repeated texture, and overexposure of the depth information in the dual-camera mode. , underexposure and other areas where the depth is wrong, which improves the accuracy of the depth information.

Further, in some embodiments, after S304, the method may further include:

The main color image is blurred according to the depth image to obtain a target image.

It should be noted that a desired target image can be obtained by performing a blurring process on the main color image according to the acquired depth image, wherein the target image may be a bokeh image. In addition, due to the high complexity of the algorithm in the embodiment of the present application, the depth image correction method is mainly used in the photographing mode of double-camera portrait blurring, and cannot be applied in the preview mode. Specifically, the embodiments of the present application are mainly aimed at applications such as blurring and matting, and optimize the depth information of the dual-camera portrait by combining the advantages of TOF. In this way, since the target image is obtained by blurring the main color image according to the acquired depth image, the target image has been repaired for no texture, repeated texture, overexposure and underexposure of the depth information in the dual-camera portrait mode. The phenomenon of depth errors in other areas improves the accuracy of depth in dual-camera portrait mode, thereby optimizing the accuracy of portrait blurring.

This embodiment provides a depth image correction method, which is applied to a terminal device. Obtain the original image corresponding to the target object and the main color image and the sub-color image corresponding to the target object; wherein, the original image is obtained according to the acquisition of the target object by the TOF sensor, and the main color image and the sub-color image are According to the acquisition of the target object by the dual cameras; then according to the main color image and the secondary color image, use a preset dual-camera algorithm to determine the first depth information and the first confidence level corresponding to the target object; the original image, and determine the second depth information corresponding to the target object; and then determine the erroneous data in the first depth information based on the first depth information, the second depth information and the first confidence level Finally, the error data area is corrected and processed through the second depth information and the main color image to obtain target depth information, and a depth image is obtained according to the target depth information; in this way, since the second depth information is composed of The first depth information is obtained from the TOF mode, and the first depth information is obtained from the dual-camera mode. The second depth information is used to optimize the first depth information, which can repair the depth information in the dual-camera portrait mode. The phenomenon of depth error in areas such as exposure, thus realizing the optimization of TOF for dual-camera portrait mode and improving the accuracy of depth in dual-camera portrait mode; in addition, the target depth information is mainly used for blurring the main color image. It can also optimize the accuracy of portrait blur, and improve the effect of portrait blur.

In another embodiment of the present application, due to the lens precision and process of the camera, distortion will be introduced to cause image distortion; and in the dual-camera mode, the optical axes of the main camera and the sub-camera are not parallel; at this time, the calculation of the dual-camera Before obtaining the first depth information in the mode, distortion correction and epipolar correction processing also need to be performed on the main color image and the sub-color image. Therefore, in some embodiments, for S302, the first depth information and the first confidence level corresponding to the target object are determined according to the primary color image and the secondary color image by using a preset dual-camera algorithm , which can include:

S302a: Perform distortion correction processing on the main color image to obtain a corrected main color image;

S302b: Perform distortion correction and epipolar correction processing on the secondary color image to obtain a corrected secondary color image;

It should be noted that the imaging process of the main camera or the sub-camera is actually a process of converting the coordinate points of the world coordinate system to the main camera coordinate system. Because the lens precision and craftsmanship of the camera will introduce distortion (the so-called distortion, specifically, the straight line in the world coordinate system will no longer be a straight line when converted to other coordinate systems), resulting in image distortion. The image is distortion corrected. In addition, in order to realize that the optical axes of the main camera and the sub-camera are completely parallel, so that the same pixel of the target object has the same height in the main color image and the sub-color image, it is also necessary to perform epipolar correction on the sub-color image. For example, Bouguet can be used. Polar correction algorithm. Specifically, since the optical axes (also called baselines) between the main camera and the sub-camera before correction are not parallel, the goal of epipolar correction is that the optical axes between the main camera and the sub-camera are completely parallel; thus , after distortion correction and epipolar correction, the images of binocular vision can be paralleled according to the same Field of Vision (FOV) standard.

It should also be noted that after acquiring the primary color image and the secondary color image, it can also be scaled according to the preset ratio to obtain a low-resolution color image; then the low-resolution color image can be obtained according to the calibration parameters corresponding to the dual cameras. The corrected color image is subjected to distortion correction and epipolar correction processing to obtain the corrected primary color image and the corrected secondary color image. The calibration parameters corresponding to the dual cameras may be obtained by calculation according to a preset calibration algorithm, such as Zhang Zhengyou's calibration method, or may be obtained directly from the manufacturer or supplier of the dual cameras. In addition, the preset ratio is a ratio value preset according to the target resolution. In practical applications, the preset ratio is set according to the actual situation, which is not specifically limited in the embodiment of the present application.

Since the polar line between the main camera and the sub-camera set on the terminal device is not parallel, the height of the same pixel in the target object in the main color image is inconsistent with the height of the pixel in the sub-color image, and After epipolar correction, the height of the same pixel of the target object in the main color image is consistent with the height in the sub-color image. In this way, when stereo matching is performed between the primary color image and the secondary color image, it is only necessary to search for matching pixels on the same row.

Exemplarily, referring to FIG. 5 , it shows a schematic diagram of comparison of effects of a polar line correction provided by an embodiment of the present application. In Figure 5, before the epipolar correction is performed on the main camera and the sub-camera, the optical axes of the two are not parallel, as shown in (a); at this time, for pixel 1 of the target object, in the obtained main color image is inconsistent with the height of the secondary color image, as shown in (b); in order to be able to calculate the first depth information, it is necessary to perform epipolar correction on the main camera and the secondary camera, so that after epipolar correction, the main camera and The optical axis of the secondary camera is completely parallel, as shown in (c); at this time, the height of pixel 1 in the main color image is the same as the height of pixel 1 in the secondary color image, as shown in (d) In this way, when the main color image and the sub-color image are stereo matched, it is only necessary to search for matching pixels on the same line, thereby greatly improving the efficiency.

S302c: For each pixel in the target object, based on the corrected primary color image and the corrected secondary color image, use a preset dual-camera algorithm to determine the first depth information corresponding to each pixel and the corresponding depth information for each pixel A first confidence level; wherein, the first confidence level is used to represent the accuracy of the first depth information.

It should be noted that, after the corrected main color image and the corrected sub-color image are obtained, the first color corresponding to each pixel in the target object can be determined according to the corrected main color image and the corrected sub-color image. Depth information and a first confidence level; wherein, the first depth information and the first confidence level are in units of pixels.

Further, in some embodiments, for S302c, determining the first depth information corresponding to each pixel based on the corrected primary color image and the corrected secondary color image may include:

The parallax matching calculation is performed on the corrected primary color image and the corrected secondary color image through the dual-camera matching algorithm, and the parallax value corresponding to each pixel is obtained;

Depth conversion is performed on the disparity value by using a first preset conversion model to obtain first depth information corresponding to each pixel.

It should be noted that the dual-camera matching algorithm is a preset algorithm or model for parallax calculation, which belongs to the classical algorithm for calculating parallax in the preset dual-camera algorithm; wherein, the dual-camera matching algorithm may be semi-global matching (Semi-GlobalMatching). , SGM) algorithm, or cross-scale cost aggregation (Cross-Scale Cost Aggregation, CSCA) algorithm, etc., which are not specifically limited in this embodiment of the present application. For example, referring to FIG. 6 , it shows a schematic diagram of the effect of a dual-camera parallax calculation provided by an embodiment of the present application. In Figure 6, it can be seen that the disparity matching calculation is performed on the two images (a) and (b), and finally the disparity effect diagram shown in (c) is obtained.

In addition, the first preset conversion model is a preset model for parallax depth conversion, which usually refers to a triangulation ranging model that uses parallax values and preset imaging parameters to calculate depth information; The preset imaging parameters may include baseline distance and focal length. For example, the first preset conversion model may be Z=Baseline*focal/disparity, where Z represents depth information, Baseline represents the distance of the baseline or the optical axis, focal represents the focal length, and disparity represents the disparity value; The first preset conversion model is also not specifically limited.

进而，可得到

其中，d为视差值。因此，终端设备只需要知晓基线距离b、焦距f以及视差值d之后，就可以根据第一预设转换模型(比如

)，计算出每个像素点对应的第一深度信息。7, which shows a schematic diagram of a model for calculating depth information provided by an embodiment of the present application. As shown in Figure 7, _OR is the location of the main camera, O _T is the location of the secondary camera, the distance between _OR and O _T is the baseline distance, denoted by b; P is the location of the target object, P ₁ is the image point obtained when the terminal device collects the target object P through the main camera, P ₁ ′ is the image point obtained when the terminal device collects the target object P through the sub-camera, and x _R is the image point P _of the target object in the main camera. The coordinates in the color image, x _T is the coordinates of the image point P ₁ ' of the target object in the sub-color image, and f is the focal length between the main camera and the sub-camera. At this time, it can be seen from the similar triangle

Furthermore, it can be obtained

where d is the disparity value. Therefore, after the terminal device only needs to know the baseline distance b, the focal length f and the parallax value d, it can convert the model according to the first preset (for example,

) to calculate the first depth information corresponding to each pixel.

After the corrected primary color image and the corrected secondary color image are acquired, the first confidence level may also be determined. In the embodiment of the present application, the first confidence level may be calculated according to the matching similarity cost between the corrected main color image and the corrected secondary color image, or may be calculated according to the texture gradient difference corresponding to the primary color image and the secondary color image It can be obtained by calculation, which is not specifically limited in the embodiments of the present application.

Further, in some embodiments, for S302c, determining the first confidence level corresponding to each pixel based on the corrected primary color image and the corrected secondary color image may include:

The matching similarity calculation is performed on the corrected primary color image and the corrected secondary color image, and the matching similarity cost corresponding to each pixel is obtained;

Based on the matching similarity cost, a first confidence level corresponding to each pixel is determined.

It should be noted that, by performing matching similarity calculation on the corrected main color image and the corrected secondary color image, the matching similarity cost corresponding to each pixel in the target object can be obtained; wherein, the specific calculation of the matching similarity cost In practical application, it can be set according to the actual situation, which is not specifically limited in the embodiment of the present application.

It should also be noted that, after the matching similarity cost is obtained, the first confidence level corresponding to each pixel can be further determined according to the matching similarity cost. Specifically, the terminal device can set a cost threshold, and compare the matching similarity cost obtained by each pixel with the cost threshold to determine the first confidence level; for example, when the matching similarity cost corresponding to the pixel is greater than the cost threshold, it indicates that The corrected primary color image and the corrected secondary color image still have a high probability of false matching for this pixel, and at this time, the first confidence level corresponding to this pixel is relatively low. In addition, the matching similarity calculation is performed for each pixel, and the minimum matching similarity cost and the second-minimum matching similarity cost can be obtained; if the minimum matching similarity cost is close to the second-minimum matching similarity cost, then It indicates that the first confidence level corresponding to the pixel is low.

Further, in some embodiments, for S302c, determining the first confidence level corresponding to each pixel based on the corrected primary color image and the corrected secondary color image may include:

Calculate the first texture gradient corresponding to each pixel under the corrected main color image;

Based on the first texture gradient, a first confidence level corresponding to each pixel is determined.

It should be noted that the first confidence level may also be related to texture richness. According to the corrected main color image, the first texture gradient corresponding to each pixel can be calculated; in this way, the first confidence level can be determined according to the first texture gradient. The specific calculation method of the texture gradient may be set according to the actual situation in practical applications, which is not specifically limited in the embodiment of the present application.

This embodiment provides a depth image correction method, which is applied to a terminal device. This embodiment describes the specific implementation of the foregoing embodiments in detail, from which it can be seen that through the technical solution of this embodiment, the depth information in the dual-camera portrait mode can be repaired in the absence of texture, repeated texture, overexposure, underexposure, etc. The phenomenon that the depth of the area is wrong, thus realizing the optimization of the TOF for the dual-camera portrait mode, and improving the accuracy of the depth in the dual-camera portrait mode; in addition, the target depth information is mainly used for the blurring of the main color image. Optimize the accuracy of portrait blur and improve the effect of portrait blur.

In another embodiment of the present application, the first depth information is calculated based on the main color image and the sub-color image, and the obtained first depth information is originally in the main camera coordinate system, so there is no need to re-calculate the first depth information. The coordinate system conversion is performed for one depth; the second depth information is calculated based on the original image, and the obtained second depth information is in the TOF coordinate system, so it is also necessary to perform coordinate system conversion on the second depth to convert the It is aligned to the main camera coordinate system. Therefore, in some embodiments, for S302, the determining, according to the original image, the second depth information corresponding to the target object may include:

S302d: According to the original image, obtain the initial depth information of each pixel in the target object in the TOF coordinate system;

It should be noted that the target object is collected by the TOF sensor, and the original image (such as a set of RAW images) corresponding to the target object is obtained; in this way, the depth information of the target object can be calculated according to the original image. The initial depth information of each pixel in the TOF coordinate system; wherein, the calculation method of the initial depth information may adopt the four-phase method.

S302e: Perform coordinate system transformation on the initial depth information by using a second preset transformation model to obtain second depth information of each pixel in the main camera coordinate system.

It should be noted that the second preset transformation model is a preset model for coordinate system transformation, for example, transforming the coordinates in the TOF coordinate system into the main camera coordinate system. In this way, the initial depth information in the TOF coordinate system can be converted into the second depth information in the main camera coordinate system according to the second preset conversion model, so as to achieve pixel alignment.

Further, in some embodiments, before S302e, the method may further include:

According to the preset calibration algorithm, the TOF sensor and the dual camera are calibrated to obtain the calibration parameters;

Correspondingly, performing coordinate system transformation on the initial depth information through the second preset transformation model to obtain the second depth information of each pixel in the main camera coordinate system may include:

Based on the calibration parameters and the second preset conversion model, the initial depth information is converted into the main camera coordinate system to obtain second depth information of each pixel in the main camera coordinate system.

It should be noted that, before performing pixel alignment, it is also necessary to perform dual camera calibration on the TOF sensor and dual camera. The calibration parameters may be obtained by calculation according to a preset calibration algorithm, such as Zhang Zhengyou's calibration method, or may be obtained directly from the manufacturer or supplier of the dual camera, which is not specifically limited in the embodiments of the present application.

In this way, after the calibration parameters are obtained, the initial depth information can be converted into the main camera coordinate system according to the calibration parameters and the second preset conversion model, and the second depth information of each pixel in the main camera coordinate system is obtained, Thus, the pixel alignment of the first depth information and the second depth information is realized, which is beneficial to the subsequent fusion of the first depth information and the second depth information, so as to realize the correction of the erroneous data area in the first depth information. It should be noted that the resolution of the main camera is usually larger, and the resolution of the depth information generated by it is generally larger than the resolution of the depth information generated by the TOF; in this way, after the initial depth information is aligned to the main camera coordinate system, the obtained second depth The pixels in the information are sparse, which is beneficial to provide some sparse effective pixels for the subsequent depth information fusion.

This embodiment provides a depth image correction method, which is applied to a terminal device. This embodiment describes the specific implementation of the foregoing embodiments in detail, from which it can be seen that through the technical solution of this embodiment, the depth information in the dual-camera portrait mode can be repaired in the absence of texture, repeated texture, overexposure, underexposure, etc. The phenomenon that the depth of the area is wrong, thus realizing the optimization of the TOF for the dual-camera portrait mode, and improving the accuracy of the depth in the dual-camera portrait mode; in addition, the target depth information is mainly used for the blurring of the main color image. Optimize the accuracy of portrait blur and improve the effect of portrait blur.

In yet another embodiment of the present application, as for the erroneous data region in the first depth information, the erroneous data region is usually located in a low confidence region of the first depth information, and it can be determined by comparing the first depth information and the second depth information. The difference between the information is judged to be specifically determined. Therefore, in some embodiments, for S303, the erroneous data region in the first depth information is determined based on the first depth information, the second depth information and the first confidence level, Can include:

S303a: Determine a low-confidence region in the first depth information according to the first confidence;

It should be noted that the low confidence region in the first depth information may be determined by the first confidence level, and the first confidence level is related to the matching similarity cost and texture richness. That is, the low-confidence region in the first depth information can be determined according to the matching similarity cost between the pixels in the first depth information and the pixels in the second depth information, or the texture gradient of the pixels in the first depth information can be determined. The low-confidence region in the first depth information is determined with the texture gradient of the pixels in the second depth information.

In addition, it is assumed that the first confidence level threshold is a judgment value used to measure whether the first confidence level belongs to a low confidence level; in this way, the judgment can also be made according to the pre-acquired first confidence level threshold. Specifically, the first confidence level is compared with the first confidence level threshold; when the first confidence level is smaller than the first confidence level threshold, it indicates that the pixel corresponding to the first confidence level belongs to the low confidence level area, so as to This can result in low confidence regions in the first depth information.

S303b: for each pixel to be judged in the low confidence area, calculate the difference between the first depth information corresponding to each pixel to be judged and the second depth information corresponding to the effective neighborhood of the pixel to be judged;

It should be noted that the effective neighborhood in the second depth information refers to the area closest to the pixel to be judged, and the size of the effective neighborhood is limited, for example, it can be 5*5 or 7*7; but effective The size of the neighborhood may be set according to actual conditions in practical applications, which is not specifically limited in this embodiment of the present application.

In this way, there are valid pixels associated with the second depth information in the valid neighborhood, so that the difference between the first depth information corresponding to the pixel to be determined and the second depth information corresponding to the valid pixel can be calculated, The difference between the two is obtained; it is convenient to further determine whether the pixel to be judged is an error point according to the size of the difference.

S303c: Compare the difference with a preset difference threshold;

It should be noted that the preset difference threshold is a preset judgment value used to measure whether the pixel to be judged is an error point. In this way, after step S303c, according to the comparison result between the difference and the preset difference threshold, when the difference is greater than the preset difference threshold, execute step S303d; when the difference is not greater than the preset difference threshold, execute step S303e .

S303d: when the difference is greater than a preset difference threshold, mark the pixel to be judged as an error point, and obtain an error data area in the first depth information according to the marked error point;

S303e: When the difference is not greater than a preset difference threshold, retain the first depth information corresponding to the pixel to be determined, and obtain a reserved data area in the first depth information.

It should be noted that, in the dual-camera mode, the first depth information has a depth error in areas without texture, repeated textures, etc. At this time, in order to improve the accuracy of the depth in the dual-camera mode, the embodiment of the present application needs to determine the depth error. The error data area is used to distinguish the error data area from the original data area for the first depth information. Specifically, after both the first depth information and the second depth information are aligned in the main camera coordinate system, according to the first confidence, a low confidence region in the first depth information can be determined; then in the low confidence region , through the first depth information corresponding to the pixel to be judged and the second depth information corresponding to the effective neighborhood of the pixel to be judged, the difference between the two can be calculated; the difference is compared with the preset difference threshold; When the value is greater than the preset difference threshold, the to-be-determined pixel can be marked as an error point, and according to the marked error point, the error data area in the first depth information is obtained; when the difference is not greater than the preset difference threshold, Indicates that the first depth information corresponding to the pixel to be judged is correct. At this time, it is necessary to retain the first depth information corresponding to the pixel to be judged, and obtain the reserved data area in the first depth information, that is, the first depth information. The original data area in the first depth information is divided into the error data area and the original data area.

Further, in some embodiments, the method may also include:

S303f: If there is no valid pixel point associated with the second depth information in the valid neighborhood of the pixel to be determined, the step of comparing the difference with a preset difference threshold is not performed.

It should be noted that since the initial depth information generated by TOF is aligned with the main camera coordinate system, the pixels in the obtained second depth information are sparse; thus, for the second depth information, the pixels to be judged may be within the effective neighborhood of the pixel. There may be no valid pixels associated with the second depth information. That is to say, when there is no valid pixel corresponding to the second depth information in the valid neighborhood of the pixel to be judged, the step of comparing the difference with the preset difference threshold cannot be performed at this time, and the corresponding pixel to be judged can be retained. The first depth information of .

Further, in some embodiments, for S304, performing correction processing on the erroneous data area by using the second depth information and the main color image to obtain target depth information may include:

S304a: For each error point in the error data area, perform weighted interpolation calculation on the error data area according to the second depth information corresponding to each error point, and replace the first depth with the calculated depth information The wrong data area in the information, get new depth information;

S304b: Perform filtering processing on the new depth information according to the main color image to obtain the target depth information.

It should be noted that, for the error points contained in the error data area, the second depth information corresponding to the effective pixel points in the effective neighborhood can be used, and at the same time, the weight of the color similarity and the spatial distance can also be used to weight the error data area. Interpolation calculation, at this time, the calculated depth information can be obtained; then use the calculated depth information to replace the wrong data area in the first depth information, and new depth information can be obtained; it can be seen that the new depth information is determined by the first depth information. The depth information and the second depth information are fused; in order to reduce the traces of artificial synthesis, the main color image can also be used as a guide to filter the new depth information, thereby weakening the traces of artificial synthesis, according to the output target depth The information can obtain the depth image, which realizes the correction processing of the erroneous data area.

It should also be noted that the filtering processing methods include Guide Filter, DT Filter (DomainTransform Filter), and Weight Medium Filter (Weight Medium Filter). Examples are not specifically limited.

Exemplarily, see FIG. 8 , which shows a detailed schematic flowchart of a depth image correction method provided by an embodiment of the present application. As shown in Figure 8, the terminal device collects the original image through the TOF sensor, which can be composed of a set of RAW images; the terminal device collects the main color image through the main camera, and the terminal device collects the sub-color image through the sub-camera; The color image is subjected to distortion correction processing, and the secondary color image is subjected to distortion correction and epipolar correction processing to obtain the corrected primary color image and corrected secondary color image respectively; Perform matching processing with the corrected secondary color image to obtain the first depth information and the first confidence level; then perform depth information calculation on the original image to obtain the initial depth information; since the initial depth information is in the TOF coordinate system, it is also It needs to be converted into the main camera coordinate system, so as to obtain the second depth information, and realize the pixel alignment of the second depth information and the first depth information; according to the first confidence degree, determine the low confidence degree in the first depth information area, and then calculate the difference between the first depth information and the second depth information in the low confidence area; then judge whether the difference is greater than the preset difference threshold, that is, judge the safety of the difference; when the difference is greater than When the difference threshold is preset, it indicates that the difference is unsafe. At this time, the pixel to be judged is marked as an error point, and the error data area in the first depth information is obtained, and then the second depth information is used for the error data area. Correction processing (such as repair, filling and interpolation processing, etc.); when the difference is not greater than the preset difference threshold, it indicates that the difference is safe, and the first depth information corresponding to the pixel to be judged can be retained at this time; The two are fused to output the final depth image.

After the depth image is acquired, the main color image can also be blurred according to the depth image, which can improve the accuracy of portrait blurring. Referring to FIG. 9 , it shows a comparative schematic diagram of a portrait blur effect provided by an embodiment of the present application. As shown in Figure 9, both (a) and (b) are backgrounds with repetitive texture areas, where (a) provides the portrait blur effect in the dual-camera mode, and (b) provides the dual-camera mode + TOF mode. It can be seen that the portrait blur effect in the dual camera mode + TOF mode is better.

This embodiment provides a depth image correction method, which is applied to a terminal device. This embodiment describes the specific implementation of the foregoing embodiments in detail, from which it can be seen that through the technical solution of this embodiment, the depth information in the dual-camera portrait mode can be repaired in the absence of texture, repeated texture, overexposure, underexposure, etc. The phenomenon that the depth of the area is wrong, thus realizing the optimization of the TOF for the dual-camera portrait mode, and improving the accuracy of the depth in the dual-camera portrait mode; in addition, the target depth information is mainly used for the blurring of the main color image. Optimize the accuracy of portrait blur and improve the effect of portrait blur.

In still another embodiment of the present application, since the effect of the TOF mode is poor outdoors, there are a large number of holes in the second depth information, which is not suitable for performing the depth image correction method of the embodiment of the present application. Therefore, in some embodiments, after determining the second depth information corresponding to the target object according to the original image, the method further includes:

According to the original image, a second confidence level corresponding to the target object is determined; wherein, the second confidence level is used to characterize the accuracy of the second depth information;

determining the number of holes in the second depth information based on the second confidence;

If the number of holes is greater than the preset hole threshold, the method for correcting the depth image is not executed.

It should be noted that since the effect of the TOF mode is poor outdoors, the number of holes or the judgment of the hole rate can be increased at this time. If there are a large number of holes or low confidence areas in the second depth information in the TOF mode, at this time, the depth image correction method (specifically, the first depth information and the second depth information) described in this embodiment of the present application may not be executed. Fusion), still using the normal dual-camera mode to obtain depth images. Specifically, assuming that the preset hole threshold is a judgment value used to measure whether the number of holes is too large, then the number of holes in the second depth information can be determined according to the second confidence level; if the number of holes is greater than the preset number of holes The threshold value indicates that the number of holes is too large. In this case, the depth image correction method of the embodiment of the present application may not be performed, and only the normal dual-camera mode is used to obtain the depth image.

In addition, the resolution of the depth information generated by the TOF sensor in the terminal device is low, which is mainly used to correct the depth error in relatively large areas; however, for scenes with hollow and very rich depth levels, this does not have a good correction effect. Therefore, the depth image correction method according to the embodiment of the present application is mainly applied to the use in the dual-camera portrait mode.

This embodiment provides a depth image correction method, which is applied to a terminal device. This embodiment describes the specific implementation of the foregoing embodiments in detail, from which it can be seen that through the technical solution of this embodiment, the depth information in the dual-camera portrait mode can be repaired in the absence of texture, repeated texture, overexposure, underexposure, etc. The phenomenon that the depth of the area is wrong, thus realizing the optimization of the TOF for the dual-camera portrait mode, and improving the accuracy of the depth in the dual-camera portrait mode; in addition, the target depth information is mainly used for the blurring of the main color image. Optimize the accuracy of portrait blur and improve the effect of portrait blur.

Based on the same inventive concept of the foregoing embodiments, see FIG. 10 , which shows a schematic structural diagram of the composition of another terminal device 100 provided by an embodiment of the present application. As shown in FIG. 10, the terminal device 100 may include: an acquisition unit 1001, a determination unit 1002 and a correction unit 1003, wherein,

The acquiring unit 1001 is configured to acquire the original image corresponding to the target object and the main color image and the sub-color image corresponding to the target object; wherein, the original image is obtained according to the acquisition of the target object by the TOF sensor, and the The main color image and the sub-color image are obtained according to the acquisition of the target object by the dual cameras;

The determining unit 1002 is configured to use a preset dual-camera algorithm to determine the first depth information and the first confidence level corresponding to the target object according to the primary color image and the secondary color image; according to the original image, determining second depth information corresponding to the target object; and further configured to determine an erroneous data region in the first depth information based on the first depth information, the second depth information and the first confidence level ;

The correction unit 1003 is configured to perform correction processing on the erroneous data area by using the second depth information and the main color image to obtain target depth information, and obtain a depth image according to the target depth information.

In the above solution, referring to FIG. 10 , the terminal device 100 may further include a blurring unit 1004 configured to perform blurring processing on the main color image according to the depth image to obtain a target image.

In the above solution, referring to FIG. 10 , the terminal device 100 may further include a collection unit 1005, configured to collect the target object through the TOF sensor, and obtain the original image corresponding to the target object; The camera collects the target object, and obtains the main color image corresponding to the target object under the main camera and the sub-color image corresponding to the target object under the sub-camera; wherein, the dual cameras include a main camera and a sub-camera .

In the above solution, the correction unit 1003 is further configured to perform distortion correction processing on the primary color image to obtain a corrected primary color image; and perform distortion correction and epipolar correction processing on the secondary color image to obtain Corrected secondary color image;

The determining unit 1002 is specifically configured to, for each pixel in the target object, use a preset dual-camera algorithm to determine the first depth corresponding to each pixel based on the corrected primary color image and the corrected secondary color image. information and a first confidence level corresponding to each pixel; wherein, the first confidence level is used to represent the accuracy of the first depth information.

In the above solution, referring to FIG. 10 , the terminal device 100 may further include a calculation unit 1006 and a conversion unit 1007, wherein,

The computing unit 1006 is configured to perform parallax matching calculation on the corrected primary color image and the corrected secondary color image through a dual-camera matching algorithm to obtain a parallax value corresponding to each pixel;

The conversion unit 1007 is configured to perform depth conversion on the disparity value by using a first preset conversion model to obtain first depth information corresponding to each pixel.

In the above solution, the calculation unit 1006 is further configured to perform matching similarity calculation on the corrected primary color image and the corrected secondary color image, to obtain the matching similarity cost corresponding to each pixel;

The determining unit 1002 is further configured to determine a first confidence level corresponding to each pixel based on the matching similarity cost.

In the above solution, the calculating unit 1006 is further configured to calculate the first texture gradient corresponding to each pixel under the corrected main color image;

The determining unit 1002 is further configured to determine a first confidence level corresponding to each pixel based on the first texture gradient.

In the above solution, the conversion unit 1007 is further configured to obtain the initial depth information of each pixel in the target object in the TOF coordinate system according to the original image; The initial depth information is converted into a coordinate system to obtain the second depth information of each pixel in the main camera coordinate system.

In the above solution, the computing unit 1006 is further configured to perform calibration between the TOF sensor and the dual cameras according to a preset calibration algorithm to obtain calibration parameters;

The conversion unit 1007 is specifically configured to convert the initial depth information into the main camera coordinate system based on the calibration parameters and the second preset conversion model to obtain the second depth of each pixel in the main camera coordinate system information.

In the above solution, referring to FIG. 10 , the terminal device 100 may further include a judgment unit 1008, wherein:

The determining unit 1002 is further configured to determine a low-confidence region in the first depth information according to the first confidence;

The calculating unit 1006 is further configured to, for each pixel to be judged in the low confidence region, calculate the first depth information corresponding to each pixel to be judged and the second depth corresponding to the effective neighborhood of the pixel to be judged the difference between the information;

The judging unit 1008 is configured to compare the difference with a preset difference threshold; and when the difference is greater than the preset difference threshold, mark the pixel to be judged as an error point, according to the marked error point to obtain the erroneous data area in the first depth information.

In the above solution, the judging unit 1008 is further configured to retain the first depth information corresponding to the pixel to be judged when the difference is not greater than a preset difference threshold, and obtain the reserved first depth information data area.

In the above solution, the judging unit 1008 is further configured to, if there is no valid pixel point associated with the second depth information in the valid neighborhood of the pixel to be judged, not to perform the step of comparing the difference value with The step of comparing with preset difference thresholds.

In the above solution, the correcting unit 1003 is specifically configured to, for each error point in the error data region, perform a weighted interpolation calculation on the error data region according to the second depth information corresponding to each error point, and Replace the wrong data area in the first depth information with the calculated depth information to obtain new depth information; and perform filtering processing on the new depth information according to the main color image to obtain the target depth information.

In the above solution, the determining unit 1002 is further configured to determine, according to the original image, a second confidence level corresponding to the target object; wherein the second confidence level is used to represent the depth of the second depth information. accuracy; and based on the second confidence, determining the number of holes in the second depth information;

The judging unit 1008 is further configured to not execute the method for correcting the depth image if the number of holes is greater than a preset hole threshold.

It can be understood that, in this embodiment, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, it may also be a module, and it may also be non-modular. Moreover, each component in this embodiment may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software function modules.

If the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or The part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, removable hard disk, Read Only Memory (ROM), Random Access Memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Therefore, the present embodiment provides a computer storage medium, where the computer storage medium stores a depth image correction program, and when the depth image correction program is executed by at least one processor, implements any one of the foregoing embodiments. method.

Based on the composition structure and computer storage medium of the above-mentioned terminal device 100, see FIG. 11, which shows the specific hardware structure of the terminal device 100 provided by the embodiment of the present application, which may include: a communication interface 1101, a memory 1102, and a processor 1103; each Components are coupled together through a bus system 1104 . It will be appreciated that the bus system 1104 is used to implement the connection communication between these components. In addition to the data bus, the bus system 1104 also includes a power bus, a control bus, and a status signal bus. However, for clarity of illustration, the various buses are labeled as bus system 1104 in FIG. 11 . Among them, the communication interface 1101 is used for receiving and sending signals in the process of sending and receiving information with other external devices;

a memory 1102 for storing computer programs that can be executed on the processor 1103;

The processor 1103 is configured to, when running the computer program, execute:

Obtain the original image corresponding to the target object and the main color image and the sub-color image corresponding to the target object; wherein, the original image is obtained according to the acquisition of the target object by the TOF sensor, and the main color image and the sub-color image are Obtained according to the acquisition of the target object by the dual cameras;

According to the primary color image and the secondary color image, the preset dual-camera algorithm is used to determine the first depth information and the first confidence level corresponding to the target object; according to the original image, the first depth information corresponding to the target object is determined. 2. depth information;

determining an erroneous data region in the first depth information based on the first depth information, the second depth information and the first confidence level;

Correction processing is performed on the erroneous data area by using the second depth information and the main color image to obtain target depth information, and a depth image is obtained according to the target depth information.

It can be understood that the memory 1102 in this embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Wherein, the non-volatile memory may be Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (Erasable PROM, EPROM), Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double DataRate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) and Direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 1102 of the systems and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

The processor 1103 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 1103 or an instruction in the form of software. The above-mentioned processor 1103 may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA), or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 1102, and the processor 1103 reads the information in the memory 1102, and implements the above method in combination with its hardware.

It will be appreciated that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more Application Specific Integrated Circuits (ASIC), Digital Signal Processing (DSP), Digital Signal Processing Device (DSP Device, DSPD), programmable logic Devices (Programmable Logic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application or a combination thereof.

For a software implementation, the techniques described herein may be implemented through modules (eg, procedures, functions, etc.) that perform the functions described herein. Software codes may be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

Optionally, as another embodiment, the processor 1103 is further configured to execute the method described in any one of the foregoing embodiments when running the computer program.

Optionally, as another embodiment, the terminal device 100 may include an application processor, a main camera, a sub-camera, an infrared transmitter, and a laser transmitter; wherein, the application processor may be configured to execute the computer program when running the computer program. The method of any of the preceding embodiments.

It should be noted that, in this application, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements , but also other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined under the condition of no conflict to obtain new method embodiments.

The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain a new product embodiment.

The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (17)

1. A correction method for a depth image, wherein the method comprises: Obtain the original image corresponding to the target object and the main color image and the sub-color image corresponding to the target object; wherein, the original image is obtained according to the acquisition of the target object by the TOF sensor, and the main color image and the sub-color image are obtained. The image is obtained according to the acquisition of the target object by the dual cameras; According to the primary color image and the secondary color image, the preset dual-camera algorithm is used to determine the first depth information and the first confidence level corresponding to the target object; according to the original image, the first depth information corresponding to the target object is determined. Two depth information; wherein, the first confidence level is used to characterize the accuracy of the first depth information; determining an erroneous data region in the first depth information based on the first depth information, the second depth information and the first confidence level; Correction processing is performed on the erroneous data area by using the second depth information and the main color image to obtain target depth information, and a depth image is obtained according to the target depth information. 2. The method according to claim 1, characterized in that, after obtaining the depth image according to the target depth information, the method further comprises: The main color image is blurred according to the depth image to obtain a target image. 3. The method according to claim 1, wherein the obtaining the original image corresponding to the target object and the main color image and the sub-color image corresponding to the target object, comprising: The target object is collected by the TOF sensor, and the original image corresponding to the target object is obtained; The target object is collected by dual cameras, and the main color image corresponding to the target object under the main camera and the sub-color image corresponding to the target object under the sub-camera are obtained; wherein, the dual cameras include the main camera and the sub-camera. Secondary camera. 4 . The method according to claim 1 , wherein the first depth information and the first depth information corresponding to the target object are determined by using a preset dual-camera algorithm according to the primary color image and the secondary color image. 5 . Confidence, including: Perform distortion correction processing on the main color image to obtain a corrected main color image; Performing distortion correction and epipolar correction processing on the secondary color image to obtain a corrected secondary color image; For each pixel in the target object, based on the corrected primary color image and the corrected secondary color image, a preset dual-camera algorithm is used to determine the first depth information corresponding to each pixel and the first depth information corresponding to each pixel Confidence. 5. The method according to claim 4, wherein determining the first depth information corresponding to each pixel based on the corrected primary color image and the corrected secondary color image, comprising: The parallax matching calculation is performed on the corrected primary color image and the corrected secondary color image through the dual-camera matching algorithm, and the parallax value corresponding to each pixel is obtained; Depth conversion is performed on the disparity value by using a first preset conversion model to obtain first depth information corresponding to each pixel. 6. The method according to claim 4, wherein determining the first confidence level corresponding to each pixel based on the corrected primary color image and the corrected secondary color image, comprising: The matching similarity calculation is performed on the corrected primary color image and the corrected secondary color image, and the matching similarity cost corresponding to each pixel is obtained; Based on the matching similarity cost, a first confidence level corresponding to each pixel is determined. 7. The method according to claim 4, wherein determining the first confidence level corresponding to each pixel based on the corrected primary color image and the corrected secondary color image, comprising: Calculate the first texture gradient corresponding to each pixel under the corrected main color image; Based on the first texture gradient, a first confidence level corresponding to each pixel is determined. 8. The method according to claim 1, wherein the determining, according to the original image, the second depth information corresponding to the target object comprises: According to the original image, obtain the initial depth information of each pixel in the target object in the TOF coordinate system; The coordinate system transformation is performed on the initial depth information by using the second preset transformation model, so as to obtain the second depth information of each pixel in the main camera coordinate system. 9 . The method according to claim 8 , wherein the coordinate system conversion is performed on the initial depth information by using the second preset conversion model to obtain the second depth of each pixel in the main camera coordinate system. 10 . Before the information, the method further includes: According to the preset calibration algorithm, the TOF sensor and the dual camera are calibrated to obtain the calibration parameters; Correspondingly, performing coordinate system transformation on the initial depth information through the second preset transformation model to obtain the second depth information of each pixel in the main camera coordinate system, including: Based on the calibration parameters and the second preset conversion model, the initial depth information is converted into the main camera coordinate system to obtain second depth information of each pixel in the main camera coordinate system. 10 . The method according to claim 1 , wherein the erroneous data in the first depth information is determined based on the first depth information, the second depth information and the first confidence level. 11 . area, including: determining a low-confidence region in the first depth information according to the first confidence; For each pixel to be judged in the low confidence region, calculate the difference between the first depth information corresponding to each pixel to be judged and the second depth information corresponding to the effective neighborhood of the pixel to be judged; comparing the difference with a preset difference threshold; When the difference is greater than a preset difference threshold, the pixel to be determined is marked as an error point, and an error data area in the first depth information is obtained according to the marked error point. 11. The method according to claim 10, wherein after the comparing the difference with a preset difference threshold, the method further comprises: When the difference is not greater than the preset difference threshold, the first depth information corresponding to the pixel to be determined is retained, and the retained data area in the first depth information is obtained. 12. The method of claim 10, wherein the method further comprises: If there is no valid pixel point associated with the second depth information in the valid neighborhood of the pixel to be determined, the step of comparing the difference with a preset difference threshold is not performed. 13. The method according to claim 1, wherein, performing correction processing on the erroneous data region by using the second depth information and the main color image to obtain target depth information, comprising: For each error point in the error data area, perform weighted interpolation calculation on the error data area according to the second depth information corresponding to each error point, and use the calculated depth information to replace the first depth information , get new depth information; The new depth information is filtered according to the main color image to obtain the target depth information. 14. The method according to any one of claims 1 to 13, wherein after the second depth information corresponding to the target object is determined according to the original image, the method further comprises: According to the original image, a second confidence level corresponding to the target object is determined; wherein, the second confidence level is used to characterize the accuracy of the second depth information; determining the number of holes in the second depth information based on the second confidence; If the number of holes is greater than the preset hole threshold, the method for correcting the depth image is not executed. 15. A terminal device, characterized in that the terminal device comprises: an acquisition unit, a determination unit, and a correction unit, wherein, The acquisition unit is configured to acquire the original image corresponding to the target object and the main color image and the sub-color image corresponding to the target object; wherein, the original image is obtained according to the acquisition of the target object by the TOF sensor, and the main color image and the sub-color image corresponding to the target object are obtained; The color image and the sub-color image are obtained according to the acquisition of the target object by the dual cameras; The determining unit is configured to use a preset dual-camera algorithm to determine the first depth information and the first confidence level corresponding to the target object according to the primary color image and the secondary color image; second depth information corresponding to the target object; and further configured to determine an erroneous data region in the first depth information based on the first depth information, the second depth information and the first confidence level; Wherein, the first confidence level is used to represent the accuracy of the first depth information; The correction unit is configured to perform correction processing on the erroneous data area by using the second depth information and the main color image to obtain target depth information, and obtain a depth image according to the target depth information. 16. A terminal device, characterized in that the terminal device comprises: a memory and a processor; wherein, the memory for storing a computer program executable on the processor; The processor is configured to execute the method according to any one of claims 1 to 14 when running the computer program. 17. A computer storage medium, characterized in that the computer storage medium stores a correction program for a depth image, and when the correction program for the depth image is executed by at least one processor, any one of claims 1 to 14 is implemented. method described.

Images