CN118608388A - Super-resolution reconstruction method of Fengyun-4 satellite images based on diffusion model - Google Patents

Abstract

The present invention belongs to the field of satellite image super-resolution in computer vision, and discloses a method for super-resolution reconstruction of Fengyun-4 satellite images based on a diffusion model. The technical solution exemplified by the present invention comprises preprocessing satellite images; constructing an SSISR-DM prediction-denoising architecture comprising a GCU-UNet network and a SISR Diff diffusion model; the overall framework of SSISR-DM comprises two stages, a latent state generation stage and a reverse diffusion stage. Through the present method, a noise prediction stage is introduced on the basis of a denoising diffusion model, so that a complex task is divided into two parts, the denoising steps are reduced, the image details and clarity can be improved, the computational complexity can be reduced, and thus the processing speed, stability and robustness of the model are improved.

Description

Super-resolution reconstruction method of Fengyun-4 satellite images based on diffusion model

Technical Field

The present invention belongs to the field of satellite image super-resolution in computer vision, and in particular relates to a Fengyun-4 satellite image super-resolution reconstruction method based on a diffusion model.

Background Art

Interpolation-based methods are classic methods in the field of super-resolution, which improve image resolution by inserting new pixels between existing pixels. Common interpolation algorithms include bilinear interpolation, bicubic interpolation, and spline interpolation. Bilinear interpolation comprehensively considers the values of four adjacent pixels and generates new pixels using the linear weighted average of adjacent pixels. It is simple to calculate and fast, but it loses more details and textures. Bicubic interpolation considers a larger range of pixels and uses cubic polynomials for interpolation, which can produce smoother and more natural image effects, but there are still blurred edges and artifacts. Spline interpolation is based on the smoothness of spline functions to further improve image quality. These interpolation-based methods are widely used in many practical applications due to their high computational efficiency and simple implementation, but the details of the images they generate are often not as rich as those of learning-based methods.

Reconstruction-based methods aim to reconstruct high-resolution images from low-resolution images, emphasizing the restoration of details and textures. These methods restore images with higher resolution by combining image degradation models and prior knowledge, using multi-frame image information or through an optimization process. Typical reconstruction methods include regularization-based super-resolution (such as sparse representation and total variation regularization) and dictionary learning-based methods. These methods iteratively reconstruct high-resolution images by solving inverse problems. Compared with interpolation methods, they can better restore the edges and details of the image and reduce blur and artifacts. However, their computational complexity is high and they are more dependent on algorithm design and prior models.

SRCNN (Super-Resolution Convolutional Neural Network) is a deep learning model for image super-resolution. The network recovers high-resolution images from low-resolution images through end-to-end training. SRCNN consists of three convolutional layers: the first convolutional layer is used for feature extraction, the second convolutional layer is used for nonlinear mapping, and the third convolutional layer is used to reconstruct high-resolution images. Compared with traditional methods, SRCNN has higher restoration quality and faster processing speed, and is widely used in image enhancement, medical imaging, satellite image processing and other fields. Its simple and efficient architecture makes it a benchmark model in image super-resolution tasks.

ESPCN (Efficient Sub-Pixel Convolutional Neural Network) is a deep learning model for image super-resolution, which is characterized by efficient upsampling through sub-pixel convolution layer. Unlike traditional interpolation methods, ESPCN performs convolution operations on low-resolution images and directly converts low-resolution feature maps into high-resolution images through sub-pixel convolution in the last layer. This method not only improves computational efficiency, but also reduces memory consumption while achieving high-quality image reconstruction. ESPCN performs well in super-resolution tasks and is widely used in video processing, surveillance imaging, satellite image enhancement and other fields.

SRGAN (Super-Resolution Generative Adversarial Network) is an image super-resolution model based on generative adversarial networks (GAN). It generates high-resolution images from low-resolution images through adversarial training of the generator and the discriminator. The generator is responsible for generating realistic high-resolution images from low-resolution images, while the discriminator evaluates the authenticity of the image, prompting the generator to generate higher-quality images. The innovation of SRGAN lies in the introduction of perceptual loss, which improves the details and visual effects of the image by comparing the feature differences between the generated image and the high-resolution real image in the pre-trained convolutional neural network. SRGAN performs well in image enhancement, medical imaging, satellite image processing and other fields.

The above three types of super-resolution algorithms all have certain defects when performing super-resolution processing on satellite images. The images reconstructed by the interpolation-based method have obvious edge blurring and missing details. The larger the magnification factor, the worse the effect. The reconstruction-based method has good results but high computational complexity, slow speed, and high requirements for data scale and hardware. Although the learning-based method has significantly improved the effect, most efficient reconstruction methods have high computational complexity, rely on a large amount of high-quality training data, and have high hardware requirements, especially high-performance GPUs or dedicated hardware support. Model training is difficult, and complex models such as generative adversarial networks (GANs) are prone to instability during training, resulting in inconsistent reconstruction results.

In addition, the Fengyun-4 meteorological satellite lacks 1KM images on channels 6 and 7. The image time interval collected in the 1KM data of the visible light channel is 1 hour, and there is a lack of meteorological data for accurate prediction.

Summary of the invention

Since the Fengyun-4 meteorological satellite lacks 1KM images on channels 6 and 7, the image time interval collected in the 1KM data of the visible light channel is 1 hour, there is a lack of meteorological data for accurate prediction, and in view of the defects of the existing technology of satellite image super-resolution, the purpose of the present invention is to propose a Fengyun-4 satellite image super-resolution reconstruction method based on a diffusion model to improve image details and clarity, reduce computational complexity, and thus improve the processing speed as well as the stability and robustness of the model.

The technical solution adopted by this method of super-resolution reconstruction of Fengyun-4 satellite images based on diffusion model to solve its technical problems is:

A super-resolution reconstruction method for Fengyun-4 satellite images based on a diffusion model is provided, including:

Preprocess satellite images;

Construct an SSISR-DM prediction-denoising architecture consisting of a GCU-UNet network and a SISR Diff diffusion model;

The overall framework of SSISR-DM includes two stages: latent state generation stage and reverse diffusion stage:

The first stage is based on a U-Net network with an attention mechanism to obtain the intermediate hidden states during the diffusion process of the HR image. At the same time, a geometric correction upsampling module is added to solve the geometric distortion caused by the earth curvature and satellite sensors and quickly improve the resolution of satellite images.

The second stage is based on the denoising diffusion probability model and uses the pre-trained U-Net network to obtain the HR image from the intermediate hidden state generated in the first stage through the back diffusion process.

Furthermore, the GCU-UNet network adds a GCU geometric correction upsampling module on the basis of the UNet network. Its structure draws on the concept of sub-pixel convolutional layer and obtains high-resolution images by copying and convolving low-resolution images.

Furthermore, specifically, the GCU geometric correction upsampling module can obtain a 4X4 pixel block obtained from the features of a single pixel by replicating the LR image five times and performing convolution on each of them, and then reconstructing and filling four edge and center pixels.

Furthermore, the SISR Diff diffusion model is obtained by introducing the concept of partial denoising on the basis of DDPM, wherein the UNet network is used to predict the difference between the previous step and the next step of the diffusion step, that is, the noise.

Furthermore, the noise coefficient in the SISR Diff diffusion model is designed to grow as a quadratic function.

Compared with the prior art, the present invention has the following beneficial effects:

1. The super-resolution reconstruction method of Fengyun-4 satellite image based on diffusion model exemplified in the present invention divides the SR task into two parts, which not only takes advantage of the former part in predicting the overall characteristics and speed of the image, but also takes advantage of the diversity of details generated by the diffusion model and its robustness to noise, thereby reducing the difficulty of training and improving the overall efficiency.

2. The diffusion model-based super-resolution reconstruction method of the Fengyun-4 satellite image in the example of the present invention solves the geometric distortion problem caused by the earth curvature and sensor posture changes through the GCU-UNet network, ensuring the accuracy and detail restoration ability of the reconstructed image. Compared with ESPCN, the target of this part is high-resolution noise rather than direct HR images, so it is only necessary to retain the validity information of the LR image and obtain approximate results.

3. In the method for super-resolution reconstruction of Fengyun-4 satellite images based on the diffusion model in the example of the present invention, the pixel features of the corresponding position of the HR image after upsampling are highly correlated with the pixel and its surroundings of the corresponding LR image. Therefore, by dividing the range of the receptive field of each group of convolution kernels and distinguishing them, it can be responsible for generating pixels at fixed positions. Compared with the sub-pixel convolution layer in ESPCN, the number of parameters is further reduced, and as many LR image features as possible are retained.

4. The diffusion model-based super-resolution reconstruction method for Fengyun-4 satellite images illustrated in the present invention reduces the denoising steps and significantly reduces the computational complexity through the SISRDiff diffusion model.

5. In the FY-4 satellite image super-resolution reconstruction method based on the diffusion model exemplified in the present invention, the quadratic function has a lower test set error ratio than the square root function, which effectively suppresses the overfitting phenomenon.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG1 is a schematic diagram of the overall framework of the SSISR-DM of the present invention;

FIG2 is a schematic diagram of the GCU module structure of the present invention;

FIG3 is a schematic diagram of the GCU-UNet network structure of the present invention;

FIG. 4 is a schematic diagram of the workflow of the SSISR-DM method of the present invention.

DETAILED DESCRIPTION

The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the inventions. It should also be noted that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

Embodiment 1:

As shown in FIG1 , this embodiment provides a method for super-resolution reconstruction of Fengyun-4 satellite images based on a diffusion model, including:

Preprocess satellite images;

Construct an SSISR-DM prediction-denoising architecture consisting of a GCU-UNet network and a SISR Diff diffusion model;

The overall framework of SSISR-DM includes two stages: latent state generation stage and reverse diffusion stage:

The first stage is based on a U-Net network with an attention mechanism to obtain the intermediate hidden states during the diffusion process of the HR image. At the same time, a geometric correction upsampling module is added to solve the geometric distortion caused by the earth curvature and satellite sensors and quickly improve the resolution of satellite images.

The second stage is based on the denoising diffusion probability model and uses the pre-trained U-Net network to obtain the HR image from the intermediate hidden state generated in the first stage through the back diffusion process.

In this embodiment, the GCU-UNet network adds a GCU geometric correction upsampling module on the basis of the UNet network. Its structure draws on the concept of sub-pixel convolution layer and obtains a high-resolution image by copying and convolving a low-resolution image.

In this embodiment, specifically, the GCU geometric correction upsampling module is a schematic diagram of the GCU module structure as shown in Figure 2. Different colors in Figure 2 represent LR images after different convolutions. The LR image is copied into five parts as shown in the figure and convolved separately, and then the five-part image is reconstructed and filled with the four edge and center pixels shown, respectively, to obtain a 4X4 pixel block obtained from the features of a single pixel.

In this embodiment, the SISR Diff diffusion model is obtained by introducing the concept of partial denoising on the basis of DDPM, wherein the UNet network is used to predict the difference between the previous step and the next step of the diffusion step, that is, the noise.

In this embodiment, the noise coefficient that grows as a quadratic function is selected for the design of the noise curve in the SISR Diff diffusion model.

Through the present invention, the super-resolution reconstruction method of Fengyun-4 satellite images based on the diffusion model is used to achieve super-resolution reconstruction from 32×32 low-resolution images to 128×128 high-resolution images. The reconstructed images have more details and higher clarity, which is helpful to capture subtle surface and atmospheric features in satellite images. Experimental verification shows that this method is superior to traditional methods in terms of indicators such as mean square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and spatial correlation coefficient (CC). The present invention uses the prediction-denoising architecture of the diffusion model to first generate a high-resolution noise image and then restore it to a high-resolution image through a denoising process. This method effectively handles the one-to-many mapping problem. Experimental results show that the improved SISR-DM model has significant improvements in indicators such as RMSE, PSNR and CC compared with bilinear interpolation, indicating that the reconstructed image is clearer and has more details. By introducing the noise prediction stage on the basis of the denoising diffusion model, the complex task is divided into two parts, reducing the denoising process. The invention adopts a method for super-resolution satellite image processing and a method for super-resolution satellite image. The method can significantly reduce the computational complexity. A GCU with a geometric correction function is designed according to the characteristics of the satellite image super-resolution problem. The high-resolution image is generated by copying and convolving the low-resolution image feature map, thereby further improving the training speed and reconstruction effect of the model. The geometric distortion problem caused by the earth curvature and the satellite sensor is solved through the sub-pixel convolution layer structure. The method has broad application prospects and commercial value in the fields of meteorological forecasting, environmental protection, urban planning, disaster monitoring, etc. At the same time, when adjusting the change curve of the noise coefficient, a quadratic function and a square root function are selected for comparative tests. When the total noise ratio is around 0.54, after 1100 iterations, the test set error of the square root function is 7 percentage points higher than that of the training set, while the quadratic function is only 2 percentage points higher. Therefore, the noise coefficient that originally grows as a linear function is changed to grow as a quadratic function, which effectively suppresses the overfitting phenomenon. In addition, in the process of using the UNet network to predict noise, the invention adds a low-resolution image as a condition, which effectively improves the accuracy of noise prediction.

The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Except for the technical features described in the specification, the remaining technical features are known technologies to those skilled in the art. In order to highlight the innovative features of the present invention, the remaining technical features will not be described here in detail.

Claims (5)

1. A wind cloud number four satellite image super-resolution reconstruction method based on a diffusion model is characterized by comprising the following steps:

Preprocessing the satellite image;

Constructing a SSISR-DM prediction-denoising architecture comprising a GCU-UNet network and a SISR Diff diffusion model;

the overall framework of SSISR-DM contains two phases, the latent state generation phase and the back diffusion phase:

The first stage is based on a U-Net network added with an attention mechanism, and aims to obtain an intermediate hidden state in the HR image diffusion process, and a geometric correction up-sampling module is added at the same time, so that the problems of the earth curvature and geometric distortion caused by a satellite sensor are solved, and the resolution of a satellite image is rapidly improved;

and the second stage is based on a denoising diffusion probability model, and an HR image is obtained from the intermediate hidden state generated in the first stage through a back diffusion process by using a U-Net network trained in advance.

2. The wind cloud number four satellite image super-resolution reconstruction method based on the diffusion model according to claim 1, wherein the GCU-UNet network is characterized in that a GCU geometric correction up-sampling module is added on the basis of the UNet network, the structure of the GCU geometric correction up-sampling module is used for referencing the concept of a sub-pixel convolution layer, and a high-resolution image is obtained by copying and convoluting a low-resolution image.

3. The method for reconstructing the wind cloud number four satellite image super-resolution based on the diffusion model according to claim 2, wherein the GCU geometric correction up-sampling module is specifically configured to obtain a 4X4 pixel block obtained by single pixel point features by copying five LR images, respectively convoluting, and then reconstructing and filling four edge and center pixel points.

4. The method for reconstructing the super-resolution of the wind cloud number four satellite image based on the diffusion model according to claim 1, wherein the SISR Diff diffusion model is obtained by introducing a partial denoising concept on the basis of DDPM, and wherein the UNet network is used for predicting the difference between the previous step and the next step of the diffusion step, namely noise.

5. The method for super-resolution reconstruction of a wind cloud number four satellite image based on a diffusion model according to claim 4, wherein the noise figure growing as a quadratic function is selected from the design of a noise curve in the SISR Diff diffusion model.