CN117808860A - Image processing method, device, electronic equipment and storage medium - Google Patents

Abstract

The application discloses an image processing method, an image processing device, electronic equipment and a storage medium, wherein the image processing method comprises the following steps: acquiring an original image as a first-order image; acquiring a depth image corresponding to the first eye image; generating a target parallax image based on the depth image and the first eye image, wherein parallax exists between the target parallax image and the first eye image; and carrying out post-processing on the target parallax image to obtain a second-order image corresponding to the first-order image. The method can generate another eye image with parallax for the monocular image, so that the field of view corresponding to the image content can be expanded when the image content is displayed.

Description

Image processing methods, devices, electronic equipment and storage media

Technical field

The present application relates to the field of image processing technology, and more specifically, to an image processing method, device, electronic device and storage medium.

Background technique

With the rapid advancement of technology and living standards, electronic devices (such as smartphones, tablets, etc.) have become one of the commonly used electronic products in people's lives. Since electronic devices usually have shooting functions, people often use electronic devices to shoot photos and videos. However, in related technologies, the field of view of the content in the images captured by the electronic devices is usually limited.

Summary of the invention

The present application proposes an image processing method, device, electronic device and storage medium, which can generate another eye image with parallax for a single-eye image, thereby expanding the field of view corresponding to the image content when displaying the image content.

In a first aspect, embodiments of the present application provide an image processing method. The method includes: acquiring an original image as a first-eye image; acquiring a depth image corresponding to the first-eye image; based on the depth image and the The first eye image is used to generate a target parallax image, and there is a parallax between the target parallax image and the first eye image; the target parallax image is post-processed to obtain a second eye image corresponding to the first eye image. image.

In a second aspect, embodiments of the present application provide an image processing device, which includes: a first image acquisition module, a second image acquisition module, a parallax image generation module, and an image post-processing module, wherein the first image The acquisition module is used to acquire the original image as the first eye image; the second image acquisition module is used to acquire the depth image corresponding to the first eye image; the parallax image generation module is used to generate the disparity image based on the depth image and the The first eye image is used to generate a target parallax image, and there is a parallax between the target parallax image and the first eye image; the image post-processing module is used to post-process the target parallax image to obtain the third eye image. The second-eye image corresponding to the first-eye image.

In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the image processing method provided in the first aspect above.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores program code. The program code can be called by a processor to execute the image provided in the first aspect. Approach.

The solution provided by this application obtains the original image as the first-eye image; obtains the depth image corresponding to the first-eye image, and generates a target parallax image based on the depth image and the first-eye image. The target parallax image and the first-eye image are There is parallax between them, and the target parallax image is post-processed to obtain the second-eye image corresponding to the first-eye image. Thus, it is possible to generate another eye image with parallax based on the single-purpose image, thereby expanding the field of view corresponding to the image content when displaying the image content, and after obtaining the parallax image for the single-purpose image, the parallax image is also post-processed. , thereby improving the accuracy and quality of the obtained image of another object.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 shows a schematic flowchart of an image processing method according to an embodiment of the present application.

FIG2 is a schematic diagram showing the principles of a depth estimation model provided in an embodiment of the present application.

Figure 3 shows a schematic flowchart of an image processing method according to another embodiment of the present application.

FIG. 4 shows a schematic diagram of an affine transformation provided in an embodiment of the present application.

Figure 5 shows another schematic diagram of the affine transformation provided by the embodiment of the present application.

Figure 6 shows a schematic flowchart of an image processing method according to yet another embodiment of the present application.

FIG. 7 shows a schematic diagram of the principle of bilinear interpolation processing provided by the embodiment of the present application.

Figure 8 shows a schematic diagram of the effect provided by the embodiment of the present application.

Figure 9 shows a schematic diagram of the effect provided by the embodiment of the present application.

Figure 10 shows a schematic flowchart of an image processing method according to yet another embodiment of the present application.

Figure 11 shows a schematic flowchart of an image processing method according to yet another embodiment of the present application.

Figure 12 shows a schematic diagram of a scenario provided by the embodiment of the present application.

Figure 13 shows a block diagram of an image processing device according to an embodiment of the present application.

FIG. 14 is a block diagram of an electronic device for executing an image processing method according to an embodiment of the present application.

Figure 15 is a storage unit used to save or carry the program code for implementing the image processing method according to the embodiment of the present application.

Detailed ways

In order to enable those in the technical field to better understand the solution of the present application, the technical solution in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application.

With the development of science and technology, electronic devices are usually equipped with cameras to realize the shooting function. Currently, the popularity of electronic devices (such as smartphones, tablets, etc.) in daily life has reached nearly universal coverage. Among them, the camera module has become the main function point of electronic devices. Users can take photos through the camera function of electronic devices. With video, the resulting images are instantly captured, which is convenient and fast. Not only that, users also often upload the captured images to the Internet to share them with others.

When shooting with electronic equipment, the field of view of the image is usually limited, which means that when the captured image is subsequently used for display, it can only display the content within the limited field of view. In addition, with the application of virtual reality (VR), augmented reality (AR), and extended reality (XR) head-mounted display devices (such as smart glasses) in home scenes, there will also be electronic Linkage between the device and the head-mounted display device. When an electronic device is linked to a head-mounted display device, there is usually a scenario where the electronic device transmits the captured content to the head-mounted display device for display. In such a scenario, if the head-mounted display device directly displays the captured monocular image, Then only a 2D display effect can be achieved, and the field of view of the displayed image content is limited.

In the related art, in order to solve the problem of limited field of view of the captured monocular image, known monocular images and depth maps can be used to complete the field of view, thereby expanding the field of view of the presented image content. However, in related technologies, the accuracy and precision of visual field completion using monocular images are usually insufficient, and the image after visual field completion may have shadows, fractures, serious texture errors, etc., which will cause many problems when displayed by the device. Big error.

In response to the above problems, the inventor proposed the image processing method, device, electronic device and storage medium provided by the embodiments of the present application, which can generate another eye image with parallax based on the single-eye image, thereby expanding the image when displaying the image content The field of view corresponding to the content, and after obtaining the parallax image for the single-view image, the disparity image is also post-processed, thereby improving the accuracy and quality of the obtained other-view image. The specific image processing method will be described in detail in subsequent embodiments.

The image processing method provided by the embodiment of the present application will be introduced in detail below with reference to the accompanying drawings.

Please refer to Figure 1, which shows a schematic flowchart of an image processing method provided by an embodiment of the present application. In a specific embodiment, the image processing method is applied to the image processing device 600 shown in FIG. 13 and the electronic device 100 configured with the image processing device 600 ( FIG. 14 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device applied in this embodiment can be a smart phone, a tablet computer, an e-book, a head-mounted display device, etc., which will not be discussed here. Make limitations. The process shown in Figure 1 will be described in detail below. The image processing method may specifically include the following steps:

Step S110: Obtain the original image as the first image.

In this embodiment of the present application, the electronic device can acquire the original image and use it as the first image to generate another image based on the original image, thereby completing the field of view of the other object and expanding the field of view. Among them, the original image can be a single-purpose RGB image, or the original image can be one of the frame images in the video; the original image can be used as the first-eye image, the original image can be used as the left-eye image, or the original image can be used as the right-eye image. .

In some implementations, the electronic device can capture an image of a real scene to obtain the captured original image and use it as the first-view image. Among them, the electronic device can be a mobile terminal equipped with a camera such as a smartphone, a tablet computer, a smart watch, etc. The electronic device can collect images through a front camera or a rear camera to obtain the above original image. For example, the electronic device can pass through the rear camera. Set up the camera to collect images, and use the collected images as the above original images.

In some implementations, the electronic device can obtain the original image from a local device. That is to say, the electronic device can obtain the original image from a locally stored file. For example, when the electronic device is a mobile terminal, the original image can be obtained from a photo album, that is, the electronic device can obtain the original image from a photo album. The device collects images through the camera in advance and stores them in the local album, or downloads the images from the network and stores them in the local album, etc., and then reads the original images from the album when it is necessary to complete the field of view of the image.

In some embodiments, when the electronic device is a mobile terminal or a computer, the original image can also be downloaded from the network. For example, the electronic device can download the required original image from a corresponding server through a wireless network, a data network, etc. When the electronic device is a head-mounted display device, it can also receive images transmitted by other devices and use them as original images.

Of course, the specific way in which the electronic device obtains the original image is not limited.

Step S120: Obtain the depth image corresponding to the first eye image.

In the embodiment of the present application, after acquiring the above first eye image, the electronic device can acquire the depth image corresponding to the above first eye image, so as to generate an image having parallax with the first eye image according to the depth image corresponding to the first eye image. Among them, the depth image refers to an image that uses the distance (depth) from the image collector to each point in the scene as a pixel value. The depth image can reflect the geometric shape of the visible surface of the object, and can be calculated as point cloud data after coordinate conversion. Point cloud data with rules and necessary information can also be inversely calculated as depth image data.

In some implementations, the electronic device can input the first-eye image into a pre-trained depth estimation model to obtain a depth image output by the depth estimation model as the depth image corresponding to the first-eye image. The depth estimation model may be a depth estimation model based on a neural network, and the depth estimation model may be trained based on a large number of sample monocular images annotated with depth images.

In a possible implementation, the above depth estimation model can be trained in the following manner: obtain multiple sample monocular images and a depth image corresponding to each sample monocular image; for each sample monocular image, annotate its corresponding depth image; based on the above sample monocular images annotated with depth images, train the initial model to obtain the above depth estimation model.

Optionally, when acquiring the above multiple sample monocular images and the depth image corresponding to each sample monocular image, the binocular images collected by the binocular camera can be acquired, and one of the binocular images can be used as The above sample monocular image, for example, the left eye image or the right eye image can be used as the above sample monocular image; according to the sample monocular image and the other image corresponding to the sample monocular image, the depth image corresponding to the sample monocular image is determined .

Optionally, please refer to Figure 2. The above depth estimation model 300 can be an encoder-decoder model. The encoder 301 is used to extract image features for the input image; the decoder 302 is used to extract image features according to the encoding. The image features extracted by the machine are mapped into depth images. Among them, the encoder can be a network model such as BEiT, Swin2, Res-net, etc. The encoder can reduce the size of the feature map layer by layer through multiple convolutional layers and pooling layers to obtain contextual information of different scales and better understand The texture and shape of objects. The decoder can be a recurrent neural network (RNN) or a variational autoencoder (VAE), etc. The decoder can upsample the feature map extracted by the encoder through a skip connection (skip connection). Known as residual connection, it fuses the feature maps of different layers by directly connecting the input layer to the output layer) to obtain more detailed information to reduce the blurring of the depth map and solve the problem of gradient disappearance in the network. .

Among them, the BEiT network predicts the visual tokens (marks) of the original image based on the encoding vector of the damaged image. The network structure has two representation methods for the image, namely image patch (block) and visual token. It first splits the image into a series of patches, randomly masks the patches, generally the mask accounts for about 40%, and then flattens it into a vector and performs linear projection to obtain the patch. Coding, and perform position coding on each patch to obtain the position coding of the image; secondly, the patch coding and position coding are fused as the input of the Transformer (Transformer) layer; and then the Masked Image Modeling (MIM) layer is carried out to The patch of the above mask predicts its token representation. The network of Swin2 is modified based on the Encoder network of Transformer. The main difference between Swin2 and Swin1 is that the LN layer is post-positioned, that is, it is standardized after the residual layer is connected, which solves the problem of large differences in activation function amplitudes across layers. and the influence of training instability. Secondly, the attention mechanism will use cosine attention instead of dot product attention to avoid falling into the limit value. The Res-net network is a deep residual network that helps the network learn more complex feature representations by introducing residual connections. The residual connections allow gradients to pass through the deep network more directly while retaining input information, thus making The network can learn richer feature representations. Specifically, the residual connections in ResNet include identity mapping and residual mapping. The identity mapping transfers the input directly to the output, while the residual mapping passes through a series of convolutions. Product layers and activation functions process the input and then add it to the input to get the output. In this way, the sum of the input and the output of the residual map is the final output.

Optionally, when training the initial model based on the above sample monocular images annotated with depth images, each sample monocular image can be input to the initial model separately to obtain the estimation results corresponding to each sample monocular image. , and then determine the loss value based on the estimation results corresponding to each sample monocular image and the depth image annotated by each sample monocular image, and then adjust the model parameters of the initial model based on the calculated loss value, and then complete a epoch (round); then return to the step of inputting each sample monocular image into the initial model, obtaining the estimation result corresponding to each sample monocular image, and completing the next epoch. Repeat this to complete multiple epochs. . Among them, epoch refers to the number of times all sample monocular images are used. In layman's terms, the value of epoch is how many times the entire data set is rotated. One epoch is equal to one training time using all sample monocular images.

Optionally, you can use the Adam optimizer to iteratively update the initial model based on the loss value determined above, so that the loss value obtained each time becomes smaller until the above loss value converges, and the model at this time is saved to obtain training. The final depth estimation model. Among them, the Adam optimizer combines the advantages of two optimization algorithms, AdaGra (Adaptive Gradient, adaptive gradient) and RMSProp, to estimate the first moment of the gradient (First Moment Estimation, that is, the mean of the gradient) and the second moment of the gradient. Moment Estimation, that is, the uncentered variance of the gradient) is comprehensively considered to calculate the update step size. The training end conditions of iterative training may include: the number of iterative training reaches the target number; or the above loss value meets the set conditions. Among them, the convergence condition is to make the target loss value as small as possible, use the initial learning rate 1e-3, the learning rate decays with the cosine of the number of steps, batch_size=512, after training for multiple epochs, the convergence can be considered complete. Among them, batch_size can be understood as a batch processing parameter, and its limit value is the total number of samples in the training set; the loss value that meets the set conditions can include: the total loss value is less than the set threshold. Of course, specific training end conditions do not need to be used as limitations.

Step S130: Generate a target parallax image based on the depth image and the first eye image, and there is a parallax between the target parallax image and the first eye image.

In the embodiment of the present application, after obtaining the depth image corresponding to the above first-eye image, a target disparity image with a parallax between the first-eye image and the first-eye image can be generated based on the above first-eye image and depth image for subsequent use. Get another destination image corresponding to the first destination image. The disparity image is an image based on any image in the image pair, its size is the size of the reference image, and the element value is the disparity value.

In some implementations, considering that there is a corresponding relationship between depth and disparity under normal circumstances, for example, the corresponding relationship is: Among them, Z is the depth of each pixel, Z _max is the maximum depth existing in the depth map, and s is a random variable. Therefore, the above disparity image can be determined based on the corresponding relationship and based on the above first-eye image and depth image. Of course, the specific method of generating the target disparity image based on the above depth image and the first eye image is not limited. For example, the above disparity image can also be generated through artificial intelligence (Artificial Intelligence, AI).

Step S140: Post-process the target disparity image to obtain a second-eye image corresponding to the first-eye image.

In the embodiment of the present application, after obtaining the above target disparity image, the electronic device can also perform post-processing on the target disparity image to obtain a second-view image corresponding to the first-view image. Among them, post-processing is used to improve the accuracy and image quality of the target disparity image. Post-processing can include interpolation processing of the target disparity image, calibration using the maximum disparity value, affine transformation, etc. The specific processing included in post-processing can be Without limitation, for example, other processing for improving the accuracy and image quality of the target disparity image may also be included. Understandably, in the related art, the accuracy of the other-eye image obtained by completing the visual field of the monocular image is usually insufficient, and the image quality is poor. Therefore, the above-obtained target parallax image can be post-processed to obtain the accuracy and image A higher quality image from another camera. It should be noted that if the above first-eye image is a left-eye image, then the second-eye image is a right-eye image; if the above first-eye image is a right-eye image, then the second eye image is a left-eye image.

In some embodiments, the electronic device performs post-processing on the above target parallax image, which can be done by processing the target parallax image through an algorithm corresponding to each processing included in the post-processing; it can also be pre-trained for the above post-processing. The image processing model can be used to process the target disparity image when post-processing the above target disparity image.

The image processing method provided by the embodiment of the present application obtains the original image as the first-eye image; obtains the depth image corresponding to the first-eye image, and generates a target disparity image based on the depth image and the first-eye image. The target disparity image is the same as the first-eye image. There is parallax between the first-eye images, and the target parallax image is post-processed to obtain the second-eye image corresponding to the first-eye image. Thus, it is possible to generate another eye image with parallax based on the single-purpose image, thereby expanding the field of view corresponding to the image content when displaying the image content, and after obtaining the parallax image for the single-purpose image, the parallax image is also post-processed. , thereby improving the accuracy and quality of the obtained image of another object.

Please refer to FIG. 3 , which shows a schematic flowchart of an image processing method provided by another embodiment of the present application. This image processing method is applied to the above-mentioned electronic equipment. The process shown in Figure 3 will be described in detail below. The image processing method may specifically include the following steps:

Step S210: Obtain the original image as the first image.

Step S220: Obtain the depth image corresponding to the first eye image.

Step S230: Generate a target parallax image based on the depth image and the first eye image, and there is a parallax between the target parallax image and the first eye image.

In the embodiment of this application, the contents of steps S210 to S230 may be referred to other embodiments, and will not be described again here.

Step S240: Perform interpolation processing on the target disparity image to obtain a first disparity image.

In the embodiment of the present application, when post-processing the acquired target disparity image, interpolation processing can be performed on the target disparity image to obtain the first disparity image, so as to improve the disparity change between far and near points through non-linear transformation. Interpolation is a mathematical method used to interpolate between known data points to obtain more accurate values. In image processing, interpolation can increase the resolution of an image or improve the details of an image. Among them, the interpolation processing for the target disparity image can be cubic interpolation, bilinear difference, etc. Cubic interpolation is a higher-order interpolation method that uses the grayscale of 16 pixels around the pixel to be obtained to perform a cubic polynomial Interpolation can get better results. Bilinear interpolation uses the grayscale of four adjacent pixels of the pixel to be found to perform linear interpolation in two directions, which can get better results.

In some implementations, the electronic device may perform cubic interpolation on the target disparity image. When performing three interpolations on the target disparity image, the target disparity image can be standardized to obtain the first intermediate disparity image; the first intermediate disparity image can be subjected to the second interpolation process to obtain the second intermediate disparity image; according to the target disparity image The maximum pixel value of the second intermediate disparity image is adjusted to obtain the first disparity image. Among them, standardization processing refers to converting image data into a standardized form so that it has uniform distribution characteristics and pixel values are 0 to 1 to achieve centralized processing, thereby facilitating subsequent data analysis and processing.

In a possible implementation, the above second interpolation process can be cubic interpolation. When performing the second interpolation process on the first intermediate disparity image, the search interval can be determined according to the position of the target pixel, that is, the search interval around the pixel to be found. Pixel range; within the search interval, select the corresponding cubic polynomial to fit the grayscale change trend of the known pixel value; derive the derivative of the fitted cubic polynomial and make it zero, and solve for the extreme point, that is, the pixel to be found Possible positions; select the best pixel position according to the position of the extreme point and the corresponding gray value to obtain the best interpolation effect; then apply the best pixel position to the original image to obtain the above second intermediate disparity image.

In a possible implementation, adjusting each pixel value of the second intermediate parallax image according to the maximum pixel value in the target parallax image may be to adjust each pixel position in the second intermediate parallax image. The pixel value is multiplied by the above maximum pixel value, thereby obtaining the above first disparity image. It can be understood that the above first intermediate disparity image is an image obtained after normalization processing for the target disparity image, and its pixel value is between 0 and 1, that is, the above maximum pixel value is not greater than 1, and the pixel value at each pixel position After multiplying by the maximum pixel value, the pixel value will not be greater than 1; in addition, the pixel value in the above target disparity image represents the disparity value, so the above maximum pixel value represents the position where the disparity value is the largest, so by The pixel value at each pixel position is multiplied by the maximum pixel value above to account for the parallax change at the near point.

Step S250: performing an affine transformation on the first parallax image to obtain a second parallax image aligned with the first eye image as a second eye image corresponding to the first eye image.

In the embodiment of the present application, after interpolation processing is performed on the target disparity image to obtain the first disparity image, affine transformation can be performed on the first disparity image to achieve the transformation between the first disparity image and the first eye image. Align to obtain a second parallax image, which can be used as another eye image (that is, a second eye image) that can be displayed together with the first target image. Among them, the affine transformation of the image is to perform operations such as translation, rotation, and scaling on the image while keeping the relative positional relationship of the image content unchanged. Image alignment is an image processing technology whose purpose is to combine two or more images. Align so that the corresponding pixels can overlap. In image registration, images from different viewing angles can be aligned through affine transformation for subsequent comparison or fusion. Thus, the first-eye image and the second-eye image can be aligned. After the images are displayed together, they can be fused by the user's brain, allowing the user to see a larger image than the original field of view of the first image.

In some implementations, considering that the above first eye image may be a left eye image corresponding to the left eye, or may be a right eye image corresponding to the right eye, the second parallax image obtained above may be corresponding to the right eye, or may be a left eye image corresponding to the right eye. Correspondingly, when performing affine transformation on the first disparity image to align the first disparity image with the first eye image, forward mapping (warping) or backward mapping may be performed on the first disparity image. Wherein, if the first eye image is a left eye image, forward mapping processing can be performed on the first parallax image to obtain a second parallax image aligned with the first eye image as the second eye image corresponding to the first eye image; If the first eye image is a right eye image, backward mapping processing can be performed on the first parallax image to obtain a second parallax image aligned with the first eye image as the second eye image corresponding to the first eye image.

It can be understood that if the first eye image is a left eye image, the first parallax image corresponds to the right eye, so the first parallax image needs to be aligned to the left eye image, as shown in Figure 4, and the pixels in the first parallax image can be forward mapped; if the first eye image is a right eye image, the first parallax image corresponds to the left eye, so the first parallax image needs to be aligned to the right eye image, as shown in Figure 5, and the pixels in the first parallax image can be backward mapped.

The image processing method provided by the embodiment of the present application can generate another eye image with parallax based on the single-purpose image, thereby expanding the field of view corresponding to the image content when displaying the image content. In addition, after obtaining the parallax image for the single-purpose image , the parallax image is also interpolated, which can improve the accuracy of the obtained other-eye image, and the parallax image is also subjected to affine transformation, so that the obtained other-eye image can be aligned with the original monocular image, and then the displayed image The content can accurately display the expanded field of view content.

Please refer to FIG. 6 , which shows a schematic flowchart of an image processing method provided by yet another embodiment of the present application. This image processing method is applied to the above-mentioned electronic equipment. The process shown in Figure 6 will be described in detail below. The image processing method may specifically include the following steps:

Step S310: Acquire an original image as a first image.

Step S320: Obtain the depth image corresponding to the first eye image.

Step S330: Generate a target parallax image based on the depth image and the first eye image, and there is a parallax between the target parallax image and the first eye image.

Step S340: performing interpolation processing on the target disparity image to obtain a first disparity image.

Step S350: performing an affine transformation on the first parallax image to obtain a second parallax image aligned with the first eye image.

In this embodiment of the present application, the contents of steps S310 to S350 may be referred to other embodiments, and will not be described again here.

Step S360: Based on the occlusion position in the second parallax image, perform hole filling processing on the second parallax image, and use the second parallax image after the hole filling processing as the corresponding first eye image. Second eye image.

In the embodiment of the present application, after performing affine transformation on the first parallax image to obtain the second parallax image aligned with the first eye image, it is considered that when performing affine transformation, the second parallax image should be The areas that are visible but occluded in the first-eye image do not have corresponding pixels for mapping in the first-eye image, so holes will appear in these areas. Therefore, after obtaining the above second disparity image, it can also be based on the second disparity image. At the occlusion position, hole filling processing is performed on the second parallax image to avoid problems such as breaks, shadows, texture errors, unclear textures, and shadows in the image, thereby improving the image quality of the second parallax image.

In some embodiments, the electronic device performs hole filling processing on the second disparity image based on the occlusion position in the second disparity image, which may include: performing a first interpolation processing on the occlusion position in the second disparity image; and filling target pixel values in other areas of the second disparity image except the occlusion position.

In the above embodiments, the occlusion position in the second parallax image refers to the hole that appears due to the lack of corresponding pixels in the first image for mapping during affine transformation. When the electronic device performs hole filling processing on the second parallax image based on the occlusion position in the second parallax image, it can identify the pixels based on the pixel values of each pixel point in the second parallax image and the pixel points in the area adjacent to each pixel point. The above occlusion position can, for example, identify areas where the pixel value is black but the pixels in the adjacent area are not black. After identifying the occlusion position, other areas in the second parallax image other than the occlusion position can be determined. These other areas can be regarded as easily deformable areas. When performing hole filling processing on the second parallax image, the identified areas can be identified. The occlusion position is interpolated; the target pixel values are filled in for the above other areas in the second disparity image.

In a possible implementation, the first interpolation processing is performed on the occluded position in the second parallax image, and bilinear interpolation processing may be performed on the occluded position in the second parallax image. Bilinear interpolation processing uses bilinear polynomials to approximate the grayscale values between image pixels. Bilinear interpolation calculates the grayscale values of two pixels and calculates the grayscale value of a new pixel according to their weights. The formula for bilinear interpolation processing is as follows:

Q ₁₁ = (x ₁ , y ₁ ), Q ₁₂ = (x ₁ , y ₂ )

Q ₂₁ = (x ₂ , y ₁ ), Q ₂₂ = (x ₂ , y ₂ )

f(x,y)＝f(Q ₁₁ )(x ₂ -x)(y ₂ -y)+f(Q ₂₁ )(xx ₁ )(y ₂ -y)

+f(Q ₁₂ )(x ₂ -x)(yy ₁ )+f(Q ₂₂ )(xx ₁ )(yy ₁ )

Among them, f(x,y) represents the gray value of the target pixel, f(Q ₁₁ ), f(Q ₂₁ ), f(Q ₁₂ ) and f(Q ₂₂ ) respectively represent the four pixels that are deduced back. Gray value, (x ₂ -x) and (y ₂ -y), (xx ₁ ) and (y ₂ -y), (x ₂ -x) and (yy ₁ ), (xx ₁ ) and (yy ₁ ) respectively represent the difference between the position coordinates of the target pixel and the reversed position coordinates. Please refer to Figure 7. The principle of the above formula is that the calculation process of this formula is to perform linear interpolation in two directions, and then add the results. Specifically, for the gray value of the target pixel, first based on the back-reduction Calculate four weighted values based on the grayscale values of the four pixels and their distance from the target pixel, and then calculate the grayscale value of the target pixel based on these four weighted values and the grayscale values of the corresponding pixels. The above The process is performed once for each pixel of the above occlusion position, thereby obtaining a new gray value distribution of the occlusion position in the second disparity image.

In a possible implementation, the target pixel values are filled in the other areas in the second parallax image, which may be black pixel values for the pixels in the other areas; or the color information of the pixels in the background area adjacent to the pixels in the other areas is used to fill these areas. It can be understood that in addition to the holes in the occluded area, there are a few blank pixels, which are regarded as noise points. Therefore, the image quality can be improved through the above filling.

Exemplarily, please refer to Figure 8. Figure 8 shows an image (first-eye image) taken of an indoor scene and another eye image (second-eye image) obtained through the image processing method provided by the embodiment of the present application. Schematic diagram, it can be seen that images taken for indoor scenes can complete the field of view at the edge and obtain images with higher image quality; please refer to Figure 9, which shows images taken for outdoor scenes (first eye image), through the image processing method provided by the embodiment of the present application, the schematic diagram of the other eye image (the second eye image) obtained can be seen as an image taken for an outdoor scene, and the field of view completion at the edge can also be achieved, and Images with higher image quality can also be obtained.

The image processing method provided by the embodiment of the present application can generate another eye image with parallax based on the single-purpose image, thereby expanding the field of view corresponding to the image content when displaying the image content. In addition, after obtaining the parallax image for the single-purpose image , the parallax image is also interpolated, which can improve the accuracy of the obtained other-eye image, and the parallax image is also subjected to affine transformation, so that the obtained other-eye image can be aligned with the original monocular image, and then the displayed image The content of the expanded field of view can be accurately displayed; in addition, after affine transformation is performed on the parallax image, the hole filling process of the parallax image is also performed, thereby further improving the image quality of the final image of the other eye.

Please refer to FIG. 10 , which shows a schematic flowchart of an image processing method provided by yet another embodiment of the present application. This image processing method is applied to the above-mentioned electronic device. The process shown in Figure 10 will be described in detail below. The image processing method may specifically include the following steps:

Step S410: Obtain the original image as the first image.

Step S420: Obtain the depth image corresponding to the first eye image.

In the embodiment of the present application, the content of step S410 and step S420 can be referred to the previous embodiment, and will not be described again here.

Step S430: Input the depth image and the first eye image into a pre-trained disparity estimation model to obtain a disparity image output by the disparity estimation model as a target disparity image. The disparity estimation model is pre-trained according to the first disparity estimation model. A sample image and a depth image corresponding to the first sample image are obtained through training. The first sample image is annotated with a corresponding second sample image. The second sample image is identical to the first sample image. There is parallax between images.

In the embodiment of the present application, when generating the target disparity image based on the above depth image and the first eye image, the depth image and the first eye image can be input into a pre-trained disparity estimation model, and then the disparity estimation model can be obtained Output disparity image. The disparity estimation model may be a convolutional neural network. For example, the disparity estimation model includes an encoding network and a decoding network. After the image input to the encoding network is activated by convolution, batch normalization (BN) and activation function (Relu) , output image features, the decoding network performs convolution, batch normalization and Relu function activation on the input image features, and then outputs a disparity image after passing through multiple residual blocks and convolution layers.

In some embodiments, the disparity estimation model is trained by: obtaining a plurality of sample image pairs collected by a binocular camera; obtaining a sample image corresponding to the first eye in the plurality of sample image pairs as the first sample image, and an image corresponding to the second eye in the plurality of sample image pairs as the second sample image; for each of the first sample images, annotating the second sample image corresponding to the first sample image, and obtaining a depth image corresponding to the first sample image to obtain a sample image set; based on the sample image set, training the initial estimation model to obtain the disparity estimation model. Among them, the plurality of sample image pairs collected by the binocular camera above can be a plurality of sample image pairs obtained by taking images for different scenes; since each of the sample image pairs above is collected by a double-sided camera, there is a disparity between the two sample images in the sample image pair, that is, each sample image pair includes a left eye sample image corresponding to the left eye and a right eye sample image corresponding to the right eye, so one of the target images in the sample image pair can be used as the first sample image for input to the model, and the other target image can be used as the label of the first sample image to constrain the model to accurately generate a disparity image.

In some embodiments, training an initial estimation model based on the sample image set to obtain the disparity estimation model may include: combining the first sample image and the first sample image in the sample image set. The depth image corresponding to this image is input to the initial estimation model to obtain the disparity estimation image output by the initial estimation model; between the second sample image marked based on the first sample image and the disparity estimation image difference, determine the loss value; based on the loss value, iteratively update the initial estimation model to obtain the disparity estimation model.

In the above embodiments, when obtaining the depth image corresponding to the first sample image, the method of obtaining the depth image may be the same as the method of obtaining the depth image corresponding to the first eye image in the previous embodiment. For example, the first sample image can be input into a pre-trained depth estimation model, thereby obtaining a depth image corresponding to the first sample image output by the depth estimation model.

In a possible implementation, after inputting the first sample image and the depth image corresponding to the first sample image into the above initial estimation model, the initial estimation model can output its corresponding disparity for the first sample image. Estimating the image; then, the loss value can be determined based on the difference between the disparity estimation model output by the initial estimation model and the label corresponding to the first sample image (ie, the second sample image to which the first sample image is labeled). Optionally, L2 loss calculation can be used to determine the above loss value.

In a possible implementation, when the initial estimation model is iteratively updated based on the obtained loss value, the model parameters of the initial estimation model can be adjusted according to the calculated loss value; and then return to step: convert the sample image The concentrated first sample image and the depth image corresponding to the first sample image are input to the initial estimation model, and a disparity estimation image output by the initial estimation model is obtained. Until the training end condition is met, the disparity after training is obtained. Estimation model.

Wherein, each first sample image and its depth image are respectively input into the initial estimation model to obtain a disparity estimation image corresponding to each first sample image, and then the disparity estimation image corresponding to each first sample image is obtained. and the second sample image annotated by each first sample image, determine the loss value, and then adjust the model parameters of the initial estimation model based on the calculated loss value, that is, one epoch (round) is completed; and then return Step: Input the first sample image in the sample image set and the depth image corresponding to the first sample image into the initial estimation model, obtain the disparity estimation image output by the initial estimation model, and complete the next epoch. , repeated in this way, multiple epochs can be completed. Among them, epoch refers to the number of times that all first sample images and their depth images are used. In layman's terms, the value of epoch is how many times the entire data set is rotated. One epoch is equal to using all the first sample images and their depth images. Train once.

In some implementations, the Adam optimizer can be used to iteratively update the initial estimated model according to the loss value, so that the loss value obtained each time becomes smaller until the above loss value converges, and the model at this time is saved to obtain training. The final disparity estimation model. Among them, the Adam optimizer combines the advantages of the two optimization algorithms AdaGra and RMSProp, comprehensively considers the first-order moment estimate and the second-order moment estimate of the gradient to calculate the update step size. Among them, the training end conditions of iterative training may include: the number of iterative training reaches the target number; or the above loss value meets the set conditions.

Optionally, the convergence condition is to make the target loss value as small as possible, use the initial learning rate 1e-3, the learning rate decays with the cosine of the number of steps, batch_size=512, after training for multiple epochs, the convergence can be considered complete. Among them, batch_size can be understood as a batch processing parameter, and its limit value is the total number of training set samples.

Optionally, the loss value satisfying the set condition may include: the loss value is less than the set threshold. Of course, specific training end conditions do not need to be used as limitations.

In a possible implementation, if the required disparity estimation model is to output the disparity image corresponding to the right eye based on the input image corresponding to the left eye and its depth image, that is, the above original image is obtained as the left eye image corresponding to the left eye. The second eye image is a right eye image, then the above first sample image can be the left eye image in the sample image pair, and the second sample image can be the right eye image in the sample image pair; if the required disparity estimation model is based on the input The image corresponding to the right eye and its depth image are output, and the disparity image corresponding to the left eye is output. That is, the above original image is the right eye image corresponding to the right eye, and the second eye image obtained is the left eye image, then the above first sample image can be a sample image pair The second sample image may be the left-eye image of the sample image pair.

In a possible implementation, the above initial estimation model includes a generating network and a discriminating network. The generating network is used to output a corresponding disparity estimation image based on each input first sample image and its depth image. In other words, the initial estimation model can be a Generative Adversarial Network (GAN). The GAN model is a deep learning model that passes at least two models in the framework: Generative Model and Discriminative Model The mutual game learning between Model) produces quite good output. In the training process of GAN, the goal of the generation network is to try to generate real disparity images to deceive the discriminant network, and the goal of the discriminant network is to try to separate the pictures generated by the generative network from the real pictures. In this way, the generation network and the discriminant network A dynamic "game process" is formed between the networks. During the training process, the disparity estimation image output by the generating network can be input to the discriminating network, and the second sample image marked by the first sample image can be input to the discriminating network. From this, the discriminating network can obtain the output of the generating network The discrimination result of the disparity estimation image, and the discrimination result of the real second sample image; and then based on the discrimination result of the disparity estimation image output by the discriminant network for the generation network, and the discrimination result of the real second sample image, the discrimination result can be determined The discriminative loss value of the network.

In addition, the loss value can also be determined based on the difference between the second sample image marked by the first sample image and the disparity estimation image as the generation loss value; and then the initial estimation model is determined based on the discrimination loss value and the generation loss value. The total loss value is then iteratively trained on the initial estimation model based on the total loss value until the training end condition, and the generated network obtained at this time is used as the disparity estimation model in the embodiment of the present application.

Step S440: Post-process the target disparity image to obtain a second-eye image corresponding to the first-eye image.

In the embodiment of the present application, step S440 can refer to the contents of other embodiments and will not be repeated here.

The image processing method provided in the embodiment of the present application can generate another eye image with parallax based on a single-eye image, thereby expanding the field of view corresponding to the image content when displaying the image content, and after obtaining the disparity image for the single-eye image, the disparity image is post-processed, thereby improving the accuracy and quality of the obtained another eye image; in addition, when acquiring the disparity image, the disparity image is generated for the single-eye image through a pre-trained disparity estimation model, thereby further improving the accuracy of the obtained another eye image.

Please refer to FIG. 11 , which shows a schematic flowchart of an image processing method provided by yet another embodiment of the present application. This image processing method is applied to the above-mentioned electronic device. The process shown in Figure 11 will be described in detail below. The image processing method may specifically include the following steps:

Step S510: Obtain the original image as the first image.

Step S520: Obtain a depth image corresponding to the first image.

Step S530: generating a target disparity image based on the depth image and the first visual image, wherein there is disparity between the target disparity image and the first visual image.

Step S540: post-processing the target parallax image to obtain a second image corresponding to the first image.

In the embodiment of the present application, the contents of steps S510 to S540 can be referred to the previous embodiments, and will not be described again here.

Step S550: sending the first eye image and the second eye image to a head mounted display device, so that the head mounted display device displays the first eye image and the second eye image.

In an embodiment of the present application, the electronic device can be connected to a head-mounted display device. After acquiring the second eye image corresponding to the first eye image, the electronic device can send the first eye image and the second eye image to the head-mounted display device so that the head-mounted display device displays the first eye image and the second eye image. Thus, when the head-mounted display device displays the first eye image and the second eye image, the image content of the first eye image can enter the user's left eye, and the image content of the second eye image can enter the user's right eye. After being fused by the user's brain, the user can see an image with a larger field of view than the field of view of the first eye image displayed alone, thereby achieving an expansion of the field of view.

For example, referring to FIG. 12 , the head-mounted display device 200 may be a wireless device. When the head-mounted display device 200 displays content, the head-mounted display device 200 may be connected to the electronic device 100 . The first-eye image and the second-eye image obtained by the electronic device 100 are sent to the head-mounted display device 200, and the head-mounted display device 200 displays the first-eye image and the second eye image. The head-mounted display device 200 can be AR glasses, AR helmets, VR glasses, VR helmets, MR (MR, MixedReality) glasses, MR helmets, etc., which are not limited here.

In some implementations, the electronic device obtains the application scenario for executing the image processing method provided by the embodiments of the present application. This may be that the electronic device captures images in real time, and executes the image processing method provided by the embodiments of the present application on the captured images. That is to say, the electronic device can acquire the monocular image collected by the image acquisition device as the first-eye image; acquire the depth image corresponding to the first-eye image; and generate the target parallax image based on the depth image and the first-eye image. There is a parallax between the first-eye image and the first-eye image; the target parallax image is post-processed to obtain a second-eye image corresponding to the first-eye image; and then the first-eye image and the second-eye image are sent to the head-mounted display device. Thus, it is possible for the electronic device to generate another target image (ie, the above second-view image frame) based on the captured image (ie, the above first-view image) in real time, and send the captured image and the other target image to the head-mounted display device. to display. The above image shooting can be a photo shooting or a video shooting, which is not limited here.

In some implementations, after acquiring the above second-eye image, the electronic device may also store the first-eye image and the second-eye image. Therefore, when the first-eye image needs to be displayed through the head-mounted display device, the electronic device can send the first-eye image and the second-eye image to the head-mounted display device, so that the head-mounted display device displays the first eye image when the head-mounted display device needs to display the first eye image. , can display images with a wider field of view, thereby expanding the field of view corresponding to the image content.

In some embodiments, after obtaining the second-eye image corresponding to the above first-eye image, the electronic device can also apply the first-eye image and the second-eye image as stereoscopic data to any three-dimensional model, such as a SLAM system, to Improve the accuracy of monocular systems.

The image processing method provided by the embodiment of the present application can generate another eye image with parallax based on the single-view original image, and send the single-view original image and the generated other-view image to the head-mounted display device, and the head-mounted display device The device displays the single-purpose original image and the generated image of another purpose, thereby allowing the user to see an image with a wider field of view than that of the original image displayed alone, thus achieving an expansion of the field of view.

Please refer to FIG. 13 , which shows a structural block diagram of an image processing device 600 provided by an embodiment of the present application. The image processing device 600 applies the above-mentioned electronic equipment. The image processing device 600 includes: a first image acquisition module 610, a second image acquisition module 620, a parallax image generation module 630 and an image post-processing module 640. Among them, the first image acquisition module 610 is used to acquire the original image as the first eye image; the second image acquisition module 620 is used to acquire the depth image corresponding to the first eye image; the disparity image generation module 630 is used to generate a target parallax image based on the depth image and the first eye image, and there is a parallax between the target parallax image and the first eye image; the image post-processing module 640 is used to process the The target disparity image is post-processed to obtain a second-eye image corresponding to the first-eye image.

In some embodiments, the image post-processing module 640 may be specifically configured to: perform interpolation processing on the target disparity image to obtain a first disparity image; perform affine transformation on the first disparity image to obtain the same as the first disparity image. The second disparity image after the eye images are aligned is used as the second eye image corresponding to the first eye image.

In a possible implementation, the image post-processing module 640 may be specifically configured to: if the first eye image is a left eye image, perform forward mapping processing on the first disparity image to obtain the same image as the first disparity image. The second parallax image after eye image alignment is used as the second eye image corresponding to the first eye image; if the first eye image is a right eye image, then backward mapping is performed on the first parallax image to obtain The second disparity image aligned with the first-order image serves as the second-order image corresponding to the first-order image.

In a possible implementation, the image post-processing module 640 may be specifically configured to: perform affine transformation on the first parallax image to obtain a second parallax image aligned with the first eye image; based on the Hole filling processing is performed on the second parallax image at the occlusion position in the second parallax image, and the second parallax image after the hole filling processing is used as the second eye image.

Optionally, the image post-processing module 640 may be specifically configured to: perform a first interpolation process on the occlusion position in the second parallax image; and perform a first interpolation process on other areas in the second disparity image except the occlusion position. Fill the target pixel value.

In a possible implementation, the image post-processing module 640 may be specifically configured to: perform standardization processing on the target parallax image to obtain a first intermediate parallax image; perform second interpolation processing on the first intermediate parallax image, A second intermediate parallax image is obtained; and each pixel value of the second intermediate parallax image is adjusted according to the maximum pixel value in the target parallax image to obtain the first parallax image.

In some embodiments, the disparity image generation module 630 may be specifically configured to: input the depth image and the first eye image into a pre-trained disparity estimation model, and obtain a disparity image output by the disparity estimation model, as The target disparity image and the disparity estimation model are trained in advance based on a first sample image and a depth image corresponding to the first sample image. The first sample image is annotated with a corresponding second sample. Image, there is a parallax between the second sample image and the first sample image.

In a possible implementation, the image processing device 600 may also include an image pair acquisition module, a sample image acquisition module, an image set construction module, and a model training module. The image pair acquisition module is used to acquire multiple sample image pairs collected by the binocular camera; the sample image acquisition module is used to acquire the sample image corresponding to the first object among the multiple sample image pairs as the first sample image. , and the image corresponding to the second object among the plurality of sample image pairs, as the second sample image; the image set construction module is used to label the first sample image for each of the first sample images. The corresponding second sample image is obtained, and the depth image corresponding to the first sample image is obtained to obtain a sample image set; the model training module is used to train an initial estimation model based on the sample image set to obtain the Disparity estimation model.

Optionally, the model training module may be specifically configured to: input the first sample image in the sample image set and the depth image corresponding to the first sample image into an initial estimation model to obtain the initial estimation model. The output disparity estimation image; based on the difference between the second sample image marked by the first sample image and the disparity estimation image, determine a loss value; based on the loss value, calculate the initial estimate The model is updated iteratively to obtain the disparity estimation model.

In some embodiments, the second image acquisition module 620 may be specifically configured to: input the first mesh image to a pre-trained depth estimation model, and obtain a depth image output by the depth estimation model as the first mesh image. The depth image corresponding to the image.

In some implementations, the image processing device 600 may further include an image sending module. The image sending module is configured to send the first image and the second image to a head mounted display device, so that the head mounted display device displays the first image and the second image.

In some implementations, the first image acquisition module 610 may be specifically configured to: acquire a monocular image collected by an image acquisition device as the first image.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In several embodiments provided in this application, the coupling between modules may be electrical, mechanical or other forms of coupling.

In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software function modules.

In summary, the solution provided by this application obtains the original image as the first-eye image; obtains the depth image corresponding to the first-eye image, and generates a target disparity image based on the depth image and the first-eye image. The target disparity image is the same as the first-eye image. There is parallax between the first-eye images, and the target parallax image is post-processed to obtain a second-eye image corresponding to the first-eye image. Thus, it is possible to generate another eye image with parallax based on the single-purpose image, thereby expanding the field of view corresponding to the image content when displaying the image content, and after obtaining the parallax image for the single-purpose image, the parallax image is also post-processed. , thereby improving the accuracy and quality of the obtained image of another object.

Please refer to Figure 14, which shows a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 can be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, a head-mounted display device, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and one or more applications are configured to execute the method described in the aforementioned method embodiment.

The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

The memory 120 may include random access memory (RAM) or read-only memory (Read-Only Memory). Memory 120 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 120 may include a program storage area and a data storage area, where the program storage area may store instructions for implementing an operating system and instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing each of the following method embodiments, etc. The storage data area can also store data created during use of the electronic device 100 (such as phone book, audio and video data, chat record data), etc.

Please refer to FIG. 15 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program code is stored in the computer-readable medium 800, and the program code can be called by the processor to execute the method described in the above method embodiment.

The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: it can still Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims (15)

1. An image processing method, characterized in that the method includes: Get the original image as the first image; Acquire a depth image corresponding to the first image; Generate a target disparity image based on the depth image and the first eye image, where there is a disparity between the target disparity image and the first eye image; Post-process the target disparity image to obtain a second-eye image corresponding to the first-eye image. 2. The method according to claim 1, wherein post-processing the target disparity image to obtain a second-eye image corresponding to the first-eye image includes: Perform interpolation processing on the target disparity image to obtain a first disparity image; Perform affine transformation on the first parallax image to obtain a second parallax image aligned with the first eye image as a second eye image corresponding to the first eye image. 3. The method of claim 2, wherein the first parallax image is subjected to affine transformation to obtain a second parallax image aligned with the first eye image, as the first parallax image. The second eye image corresponding to the first eye image includes: If the first eye image is a left eye image, forward mapping is performed on the first parallax image to obtain a second parallax image aligned with the first eye image as the corresponding left eye image. Second eye image; If the first eye image is a right eye image, then perform backward mapping processing on the first parallax image to obtain a second parallax image aligned with the first eye image as the corresponding parallax image of the first eye image. Second eye image. 4. The method according to claim 2, wherein the first parallax image is subjected to affine transformation to obtain a second parallax image aligned with the first eye image, as the first parallax image. The second eye image corresponding to the first eye image includes: Perform affine transformation on the first parallax image to obtain a second parallax image aligned with the first eye image; Based on the occlusion position in the second disparity image, hole filling processing is performed on the second disparity image, and the second disparity image after the hole filling processing is used as the second eye image. 5. The method according to claim 4, characterized in that the hole filling process is performed on the second disparity image based on the occlusion position in the second disparity image, comprising: Perform a first interpolation process on the occlusion position in the second disparity image; Filling other areas in the second disparity image except the occlusion position with target pixel values. 6. The method according to claim 2, characterized in that, performing interpolation processing on the target disparity image to obtain the first disparity image includes: Perform normalization processing on the target disparity image to obtain a first intermediate disparity image; Perform a second interpolation process on the first intermediate parallax image to obtain a second intermediate parallax image; According to the maximum pixel value in the target disparity image, each pixel value of the second intermediate disparity image is adjusted to obtain the first disparity image. 7. The method according to claim 1, characterized in that generating a target disparity image based on the depth image and the first eye image comprises: The depth image and the first eye image are input into a pre-trained disparity estimation model to obtain a disparity image output by the disparity estimation model as the target disparity image. The disparity estimation model is pre-trained according to the first The sample image and the depth image corresponding to the first sample image are trained. The first sample image is annotated with the corresponding second sample image. The second sample image is the same as the first sample image. There is a parallax between them. 8. The method according to claim 7, characterized in that the disparity estimation model is trained by: Obtain multiple sample image pairs collected by binocular cameras; Acquire a sample image corresponding to a first target among the plurality of sample image pairs as the first sample image, and an image corresponding to a second target among the plurality of sample image pairs as the second sample image; For each first sample image, mark the second sample image corresponding to the first sample image, and obtain the depth image corresponding to the first sample image to obtain a sample image set; Based on the sample image set, an initial estimation model is trained to obtain the disparity estimation model. 9. The method according to claim 8, characterized in that, based on the sample image set, training an initial estimation model to obtain the disparity estimation model includes: Input the first sample image in the sample image set and the depth image corresponding to the first sample image into an initial estimation model to obtain a disparity estimation image output by the initial estimation model; Determine a loss value based on the difference between the second sample image annotated by the first sample image and the disparity estimation image; Based on the loss value, the initial estimation model is iteratively updated to obtain the disparity estimation model. 10. The method according to claim 1, wherein said obtaining the depth image corresponding to the first eye image includes: The first mesh image is input to a pre-trained depth estimation model to obtain a depth image output by the depth estimation model as the depth image corresponding to the first mesh image. 11. The method according to any one of claims 1 to 10, characterized in that after post-processing the target parallax image to obtain a second eye image corresponding to the first eye image, the method further comprises: The first eye image and the second eye image are sent to a head mounted display device, so that the head mounted display device displays the first eye image and the second eye image. 12. The method according to any one of claims 1 to 10, characterized in that obtaining the original image as the first image includes: A monocular image captured by an image acquisition device is acquired as the first monocular image. 13. An image processing device, characterized in that the device includes: a first image acquisition module, a second image acquisition module, a parallax image generation module and an image post-processing module, wherein, The first image acquisition module is used to acquire an original image as a first image; The second image acquisition module is used to acquire the depth image corresponding to the first eye image; The parallax image generation module is configured to generate a target parallax image based on the depth image and the first eye image, and there is a parallax between the target parallax image and the first eye image; The image post-processing module is used to perform post-processing on the target disparity image to obtain a second-eye image corresponding to the first-eye image. 14. An electronic device, characterized in that it includes: one or more processors; memory; one or more programs, wherein said one or more programs are stored in said memory and configured to be executed by said one or more processors, said one or more programs are configured to perform as claimed The method described in any one of 1-12. 15. A computer-readable storage medium, characterized in that program code is stored in the computer-readable storage medium, and the program code can be called and executed by a processor as described in any one of claims 1-12 Methods.