CN109035142A - A kind of satellite image ultra-resolution method fighting network integration Aerial Images priori - Google Patents

Abstract

The invention discloses a kind of satellite image ultra-resolution methods for fighting network integration Aerial Images priori, data training image super-resolution model of clearly taking photo by plane is recycled to training denoising model by using 16 grades of corresponding images constituted without noise image of 16 grades of noisy acoustic images first.Since there is no the situation of satellite image and Aerial Images pair, when carrying out post processing of image to the super resolution image of generation, using priori dictionary outside clearly Aerial Images building GMM model, and thus, the internal unsharp satellite image of guidance is rebuild.It is further promotion picture quality after reconstruction, carries out image sharpening using the mode of gaussian filtering.The full resolution pricture of protosatellite image is finally obtained, and realizes that the visual quality of images in protosatellite image basis is promoted.By experiment link it can also be seen that the validity of this programme.Satellite image super-resolution and picture quality to solve to have ready conditions under limited case in reality provide effective thinking.

Description

A satellite image super-resolution method based on adversarial network combined with aerial image prior

technical field

The invention belongs to the technical field of image super-resolution, in particular to a satellite image super-resolution method based on multi-scale perceptual loss and generation confrontation network combined with aerial image prior.

Background technique

Image resolution is an important indicator of image quality. An image with higher resolution can show more details more clearly. However, due to the influence of hardware and external environment during the image acquisition process, the acquired image resolution is lower, resulting in The problem of how to obtain high-resolution images from low-resolution images. At present, with the increase in the number of satellites, satellites can cover more than 90% of the earth, which makes the range that can be monitored by satellites much larger than the range covered by images obtained by other means, but satellite images are affected by many reasons. The rate is lower. For example, compared with aerial images, satellite images are relatively blurry and lack detailed information, but the coverage of aerial images is far less than that of satellite images, so how to obtain satellite images with higher resolution is of great significance and value.

In the field of image super-resolution, the combination of deep neural network and traditional image super-resolution problems has made a new breakthrough in image super-resolution technology. With the development of computer hardware equipment, the cost of large-scale computing acceleration has been significantly reduced, and the cost of training deep neural networks has been reduced, which greatly facilitates scientific research workers and makes this technology widely used in various fields. From the original deep learning and super-resolution network SRCNN to the current super-resolution algorithm SRGAN implemented by Generative Adversarial Nets (GAN), the network parameters are trained by using low-resolution and high-resolution images to obtain Transform models from low-resolution images to high-resolution images to generate high-resolution images when only low-resolution images are available.

The image super-resolution problem is described as follows:

The problem of image super-resolution refers to the process of obtaining a corresponding high-resolution image from a low-resolution image. Through such a technology, it breaks through the limitations of the original system imaging hardware conditions and obtains a clearer image. In the image super-resolution technology, it can generally be divided into super-resolution problems in two cases: the super-resolution method based on a single image and the super-resolution method based on multiple images. Single image super-resolution is a method of enlarging low-resolution images and improving image resolution through reconstruction algorithms. The super-resolution algorithm based on multiple images uses the method of fusion of multiple frames of similar image sequences to reconstruct high-resolution images.

In the single-image-based super-resolution method, the algorithm establishes the relationship between the low-resolution image and the high-resolution image. In this way, a high-resolution image is reconstructed from a low-resolution image. Traditional algorithms simulate the causes of low-resolution images in various ways, build various degradation models to fit the process of low-resolution image generation, and construct the relationship between low-resolution images and high-resolution images to predict the generation of high-resolution images. resolution image. Such a simulation process can be described by the following formula:

I _L =HI _H +n

Among them, I _L is the low-resolution image, I _H is the high-resolution image corresponding to I _L , H is the degradation model for generating the low-resolution image, and n is the noise interference factor in the process of generating the low-resolution image. H as a degradation model can be expressed as:

H＝D _Sub ×B×G

Among them, D _Sub represents the downsampling method, B is the blur factor, and G is the geometric deformation factor.

The methods to solve the above degradation model construction mainly include methods based on interpolation, methods based on image reconstruction, and methods based on learning. In the interpolation method, the super-resolution of the image is realized by decomposing the image, interpolating and returning the interpolation value. It has fast running speed and can be calculated in parallel, which can meet the requirements of real-time image super-resolution. But the interpolation method cannot predict the high-frequency information lost in going from low-resolution images to high-resolution images, and the resulting high-resolution images lack texture details and sharp edges. In the super-resolution algorithm based on image reconstruction, it is divided into the spatial domain method and the frequency domain method. By establishing the corresponding relationship between the low-resolution image and the high-resolution image in the spatial domain or the frequency domain, the corresponding relationship model is artificially designed to realize the The process of converting a low-resolution image to a high-resolution image. Such as the more classic convex set projection method, maximum a posteriori probability estimation, etc. The disadvantage of this method is that the artificially designed model cannot be adapted to restore a variety of image details, and the construction model can only achieve good results on a small amount of data, and cannot further improve the clarity of image details when the data increases.

In the learning-based method, similar to the image reconstruction-based method, they all establish the relationship between the low-resolution image and the high-resolution image, but the learning-based method uses external training samples to obtain information about the relationship between the low-resolution image and the high-resolution image. The prior knowledge of the relationship between high-resolution images is used to realize the transition from low-resolution images to high-resolution images. For example, methods based on manifold learning, methods based on sparse representation, and methods based on deep neural networks. In learning methods such as sparse representation, due to the size of the construction dictionary and the difficulty in guaranteeing data sparsity, stable image super-resolution results cannot be obtained. In the super-resolution method based on deep neural network, the proposed methods based on residual network and generative confrontation network need to learn and train low-resolution images and high-resolution image pairs through a large number of parameters. Such methods also exist. A large amount of data is required for training, it is easy to overfit the data during training, and it cannot obtain good robustness during testing. At the same time, when predicting high-frequency information of high-resolution images, there will still be missing, making texture-rich regions look smooth.

There are still some practical limitations in the satellite image super-resolution problem. At present, it is impossible to obtain very high-resolution satellite images, which makes it difficult to obtain high-resolution satellite image and low-resolution image pair data when performing image super-resolution. Super-resolution methods for pairs of high-resolution images and high-resolution images cannot be directly used for such satellite image super-resolution tasks. In the acquisition of satellite images, the influence of noise is serious, which makes the grain noise in the obtained image obvious. Direct super-resolution of a single image will amplify the noise in the image and affect the clarity. As auxiliary data, although the coverage of aerial images is far less than that of satellite images, there are many similarities between aerial images and satellite images, and compared with satellite images, aerial images have very good clarity. The current aerial image data and satellite image data do not have the nature of pairing, that is, they are not taken at the same place and at the same time period. How to denoise and super-resolution satellite images under the existing limited conditions and how to use clear aerial image data to enhance the definition of satellite data has become a problem to be solved.

Contents of the invention

The technical problem to be solved by the present invention is to provide a satellite image super-resolution method based on multi-scale perception loss and generative adversarial network combined with aerial image prior, which can make up for the ordinary super-resolution method that only uses satellite images. In the process of resolution algorithm, the prior insufficient problem of clear images (lack of clear satellite images) can be generated to generate clearer satellite images. At the same time, in the case of only using satellite data, due to the addition of multi-scale perceptual loss, clearer super-resolution images can be generated than other methods.

The present invention adopts following technical scheme:

An adversarial network combined with aerial image prior satellite image super-resolution method, using 16 levels of noise-containing images and their corresponding 16 levels of noise-free images to form image pairs to train the denoising model, and then using aerial photography data to train the image super-resolution model; using Aerial images construct the external prior dictionary of the GMM model, and guide the internal unclear satellite images to be reconstructed, complete the post-processing of the generated super-resolution images, and then use Gaussian filtering to sharpen the images, and finally obtain the high-resolution images of the original satellite images. Distinguish images and realize image visual quality improvement on the basis of original satellite images.

Specifically, the following steps are included:

S1. Define the generator in the generation confrontation network, the decision device and the multi-scale perceptual loss network;

S2. Use the images extracted from the 18th level in the existing satellite data to downsample to the 16th level, set the obtained 16th level satellite image as the target _{ID_H} for denoising, and the satellite data extracted from the 16th level as the noisy image _{ID_L to} form Image pair, let the generated noise-free satellite image be _{ID_GH} ;

S3. Using the image pair formed in step S2, perform initialization training on the generator in the denoising model. In the initialization training, use the mean square error as the loss function to calculate the pixels between the image generated by the generator and its corresponding target image. The mean square error of the MSE generator is obtained by the loss function loss _MSE , and the gradient is calculated and returned to adjust the model parameters;

S4. After initial training for 100 epochs, perform complete model training, calculate the loss and the corresponding gradient, and send back to adjust the parameter models in the generator and decision device. The perceptual loss network VGG19 does not adjust the parameters;

S5. According to the above settings, train 200 epochs to achieve convergence, save the model, and the trained generator is used for denoising processing. The obtained image after denoising is _{ID_GH} as the input of image super-resolution, and defines the satellite image super-resolution model;

S6. Repeat steps S3-S5 to complete the super-resolution network training process and denoising model, then generate a super-resolution image I _{SR_GH} , and construct an external prior dictionary using a Gaussian mixture model;

S7. Construct the GMM external prior dictionary, divide the clear 17-level aerial image into small blocks of 15*15, and then perform preliminary grouping according to the Euclidean distance;

S8. Reconstruct the satellite image in groups according to the reconstructed internal graphics blocks, perform an image sharpening operation on the reconstructed satellite image, and obtain a final result image.

Further, in step S1, the generator in the generative confrontation network is defined as: using a residual network as the generator, the residual network contains 16 residual modules, and each residual module contains three convolutional layers;

The decision device structure is defined as: use a 10-layer convolutional neural network as the decision device, and the convolutional layer of the convolutional neural network uses hole convolution;

The multi-scale perceptual loss is defined as: use the VGG19 network pre-trained on the IMAGENET1000 class classification database as the perceptual loss network, and construct a multi-scale perceptual loss by using _{conv2_2} , _{conv3_4} , _{conv4_4} , multi-scale feature maps in multiple layers.

Further, in step S3, the MSE generator loss function loss _MSE is as follows:

loss _MSE = MSE( _{ID_GH} , _{ID_H} ).

Further, in step S4, during model training, the MSE generator loss function loss _MSE in the generator loss function, the perceptual loss function loss _vgg and the adversarial loss function loss _GAN are weighted and added to form the generator loss function during overall training as follows:

loss _G = loss _MSE + loss _vgg + loss _GAN .

Further, the perceptual loss loss _vgg is as follows:

loss _vgg ＝10 ^-6 ×(loss _{mse_conv2_2} +loss _{mse_conv3_4} +loss _{mse_conv4_4} )

loss _{mse_conv2_2} ＝MSE(f _{i_conv2_2} , _{ft_conv2_2} )

loss _{mse_conv3_4} ＝MSE(f _{i_conv3_4} , _{ft_conv3_4} )

loss _{mse_conv4_4} ＝MSE(f _{i_conv4_4} , _{ft_conv4_4} )

Among them, f _{i_conv2_2} , f _{i_conv3_4} , f _{i_conv4_4} are input generated images to the perceptual model corresponding to _{conv2_2} , _{conv3_4} , _{conv4_4} layer feature maps, f _{t_conv2_3} , f _{t_conv3_3} , f _{t_conv4_3} are corresponding _{conv2_2} obtained in the generated image corresponding to the target image input perceptual model , _{conv3_4} , _{conv4_4} layer feature map;

The confrontation loss function loss _GAN is as follows:

loss _GAN ＝10 ^-4 ×cross_entropy( _{ID_GH} ,True)

cross_entropy(I _{D_GH} ,True)=log(D(I _{D_GH} ))

Among them, D(·) is the decision device.

Further, in step S4, the loss function loss _D of the decision device during overall training is defined as:

loss _D = loss ₁ + loss ₂

loss ₁ = sigmoid_cross_entropy( _{ID_GH} , False)

loss ₂ = sigmoid_cross_entropy( _{ID_H} , True).

Further, in step S5, the super-resolution model includes a generator, a perception model and a decision device, the perception model and the decision device have the same structure as that used in the denoising model, and the generator in the image super-resolution model is defined as follows:

By building a residual module and stacking multiple residual modules to form the main body of the network structure, the image can be enlarged and used through the sub-pixel convolution layer.

Furthermore, the data used for training the generator of the super-resolution model is aerial photography data, and the input is an image pair composed of a low-resolution 16-level aerial image of I _{SR_L} and its corresponding high-resolution 17-level aerial image I _{SR_H} , and the output of the generator is I _{SR_GH} , define the loss function of the generator as follows:

loss _{MSE_SR} = MSE(I _{SR_GH} , I _{SR_H} ).

Further, step S7 is specifically as follows:

S701. Construct a GMM model according to the grouped image blocks, perform SVD decomposition on the covariance matrix in the obtained model, and construct a dictionary as an external prior to guide the reconstruction of subsequent satellite images;

S702. Using the I _{SR_GH} output in the previous super-resolution model as the internal image input, divide the input into 15*15 blocks, and use the GMM model when constructing the external prior dictionary to guide the block clustering;

S703. Use the dictionary formed by the external prior to guide the internal image blocks to construct the internal dictionary again;

S704. Perform sparse coding on the internal dictionary, and reconstruct a new internal graphic block group by combining the original internal image block group.

Compared with the prior art, the present invention has at least the following beneficial effects:

The present invention is a satellite image super-resolution method combined with adversarial networks and aerial image priors, aiming at improving the satellite image resolution and visualization effect in real conditions, but there is no satellite image and its corresponding clear aerial image pair, it is designed A set of satellite image super-resolution process, including three parts: image denoising, image super-resolution and image post-processing, within the range of available data, gradually improve the method flow of the final satellite image super-resolution result

Furthermore, both the denoising model and the super-resolution model in the satellite image super-resolution process are composed of generative adversarial networks, and on this basis, it is proposed to add multi-scale perceptual loss to further improve the ability of generative adversarial networks to achieve image denoising and The performance of image super-resolution, the role of perceptual loss is to constrain the image generated by the generator and its corresponding target from the feature domain, so that the generated image is visually closer to the real target image. The multi-scale perceptual loss combines the perceptual loss of multiple scales and adds stronger constraints, so the generation effect has been further improved.

Furthermore, as different modules in the generative confrontation network, the generator and the decider have different roles. The definitions here play an important role in the realization of satellite image denoising and satellite image super-resolution. The generator mainly constructs losses for point-to-point pixels, and in the main body of the network, it also pays more attention to extracting high-frequency information of images (through the residual structure). The discriminator pays more attention to the high-level semantic level to ensure the consistency between the generated image and the real target image, and requires a larger receptive field (realized by hole convolution). The multi-scale perceptual loss is the constraint between the generated image and the real target image in the feature domain, which is realized by using the network pre-trained on IMAGENET.

Further, the generator generates an image that is noise-free and similar to the real clear image at the pixel level, so the loss function uses the MSE function based on the difference between pixels.

Further, the decision device constrains the similarity between the generated image and the real clear image from the high-level semantic level. Using the cross-entropy function as a loss function based on the decision probability, it is hoped that the generated image and the real target image have the highest probability of being judged as the same category semantically. That is, the generated image is as similar as possible to the real target image.

Furthermore, in the super-resolution model, the functions of the decision device and the perceptual model are the same as those in the noise model, so the same structure is used. In the generator part, the main body of the network is similar (but still needs to generate more high-frequency information, and also adopts the residual structure), but since the super-resolution model needs to generate an image larger than the input low-resolution image, sub-pixel volume is used here It is realized by the design of the combination of the accumulation layer and the grape convolution layer.

Further, in reality, it is impossible to obtain clearer satellite images (close to the definition of aerial images) and low-resolution satellite images. This problem limits the realization effect of satellite image super-resolution. The present invention proposes that after image super-resolution The image post-processing method used to further improve the image visualization effect, by using the clear aerial data GMM to build an external prior dictionary to guide the satellite image to build an internal dictionary to reconstruct a clearer satellite image.

In summary, the present invention implements the denoising model and image super-resolution model through the combination of pixel-level, semantic-level and multi-scale feature domain constraints. At the same time, for the lack of paired satellite image training data, the introduction of aerial images for image super-resolution Model training and GMM model dictionary construction in image post-processing guide the reconstruction of clearer satellite images.

The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

Description of drawings

Fig. 1 is overall flowchart;

Figure 2 is a structural diagram of the generator in the denoising model;

Fig. 3 is the structural diagram of the discriminator in the denoising model;

Fig. 4 is a structural diagram of VGG19 in the denoiser;

Fig. 5 is a generator structure diagram in the image super-resolution model;

Fig. 6 is the flowchart of constructing GMM model and guiding satellite image reconstruction using aerial images;

Fig. 7 is effect diagram of the present invention;

Fig. 8 is a comparison chart of the results of the present invention.

Detailed ways

The present invention provides a satellite image super-resolution method based on multi-scale perceptual loss and generative confrontation network combined with aerial image prior. Noise model, and then use clear aerial data to train image super-resolution model. Since there is no pair of satellite images and aerial images, when post-processing the generated super-resolution images, the clear aerial images are used to construct the external prior dictionary of the GMM model, and thus guide the internal unclear satellite images to be reconstructed . After reconstruction, in order to further improve the image quality, the Gaussian filter is used for image sharpening. Finally, the high-resolution image of the original satellite image is obtained, and the visual quality of the image is improved based on the original satellite image. The effectiveness of this scheme can also be seen from the experimental link. It provides an effective idea to solve the satellite image super-resolution and image quality improvement under the conditional constraints in reality.

Please refer to Fig. 1, the present invention is a satellite image super-resolution method based on multi-scale perception loss and generation confrontation network combined with aerial image prior, the specific steps are as follows:

S1. The generative adversarial network to realize the denoising function consists of three parts, the generator, the decision device and a VGG19 network pre-trained with the IMAGENET database;

S101. Define the generator in the generative adversarial network. Here, a residual network is used as the generator, which includes 16 residual modules, and each residual module includes three convolutional layers. The denoising function that needs to be implemented here does not need to enlarge the image here, and the specific structure is shown in Figure 2.

S102. Define the decision device structure. Here, the decision device uses a 10-layer convolutional neural network, in which the convolution layer uses hole convolution. By setting the size of the hole convolution range, the receptive field is increased without using the pooling layer. The size of the decision device improves the accuracy of the decision device. The specific structure is shown in Figure 3. The structure of the decision device contains 10 convolutional layers, and the number of convolution kernels in each layer is 64, 128, 256, 512, 1024, 512. , 256, 128, 128, 128 are arranged in increasing and decreasing patterns in turn. The convolution kernel size of the first 7 layers is 4*4, and the step size is 2. Sliding convolution is performed in turn. The increasing number of convolution kernels means As many feature types as possible. The last layer of convolution kernel uses a size of 1*1, which is used to reduce the amount of parameters. Due to the increase in the number of convolution kernels in the front, the number of channels also increases. Here, such a layer needs to be added for adjustment.

S103. Define a multi-scale perceptual loss, use the VGG19 network pre-trained on the IMAGENET1000 class classification database as the perceptual loss network, different from other perceptual losses, by using multi-scale feature maps in _{conv2_2} , _{conv3_4} , _{conv4_4} multi-layer , build a multi-scale perceptual loss, and improve the quality of the image generated by the generator. The specific structure is shown in Figure 4, which contains two convolution modules, the first one contains two convolution layers and one pooling layer, and the second convolution module Contains four convolutional layers and one pooling layer. Here, all convolution layers use 3*3 convolution kernels with a step size of 1, and the number of convolution kernels adopts a layer-by-layer increasing mode similar to that of the decision device: 64, 64, 128, 128, 256, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, 512. Among them, _{conv2_2} , _{conv3_4} , _{conv4_4} are the output of the second convolution module, the output of the third convolution module and the fourth convolution The output of the product module.

S2. Use the images extracted from the existing satellite data from level 18 to down-sample to level 16 (common satellite images are generally level 16, and the acquisition cost of level 18 data is relatively high), so the obtained level 16 data is relatively clear, but due to The acquisition cost of 18-level satellite data is very high, and such clear data is very rare.

Let the obtained 16-level satellite image be used as the target _{ID_H} for denoising, and the common satellite data directly extracted from the 16-level is used as the noisy image _{ID_L} , and the image pair is formed in this way, and the generated noise-free satellite image is set as I _{D_GH} ;

S3. Using the image pair formed in step S2, perform initialization training on the generator in the denoising model. In the initialization training, use the mean square error (MSE) as a loss function to calculate the image generated by the generator and its corresponding target image. The mean square error of the pixels between, calculate the gradient and return the adjusted model parameter loss _MSE as follows:

loss _MSE ＝MSE( _{ID_GH} , _{ID_H} )

S4, after the initial training of about 100 epochs (one epoch means that all the image data in the image library has been trained once and counted as one epoch), complete model training is carried out;

At this time, all three networks need to participate in the training, but VGG19 does not adjust the parameters, and only needs to output the perceptual loss to the generator and the decision device to adjust the parameters; during the overall training, the loss function of the generator is different from that of the individual initialization training.

During overall training, the loss function of the generator consists of three parts: MSE generator loss, perceptual loss and confrontation loss. These three parts are weighted and added to form the generator loss function during overall training:

loss _G = loss _MSE + loss _vgg + loss _GAN

Among them, loss _MSE is the same as the loss function when initializing training, and loss _vgg is the perceptual loss:

loss _vgg ＝10 ^-6 ×(loss _{mse_conv2_2} +loss _{mse_conv3_4} +loss _{mse_conv4_4} )

loss _{mse_conv2_2} ＝MSE(f _{i_conv2_2} , _{ft_conv2_2} )

loss _{mse_conv3_4} ＝MSE(f _{i_conv3_4} , _{ft_conv3_4} )

loss _{mse_conv4_4} ＝MSE(f _{i_conv4_4} , _{ft_conv4_4} )

Among them, f _{i_conv2_2} , f _{i_conv3_4} , f _{i_conv4_4} are input generated images to the perceptual model corresponding to _{conv2_2} , _{conv3_4} , _{conv4_4} layer feature maps, f _{t_conv2_3} , f _{t_conv3_3} , f _{t_conv4_3} are corresponding _{conv2_2} obtained in the generated image corresponding to the target image input perceptual model , _{conv3_4} , _{conv4_4} layer feature map;

loss _GAN is an anti-loss function:

loss _GAN ＝10 ^-4 ×cross_entropy( _{ID_GH} ,True)

cross_entropy(I _{D_GH} ,True)=log(D(I _{D_GH} ))

Among them, D(·) is the decision device.

The decision loss function for the overall training is defined as:

loss _D = loss ₁ + loss ₂

loss ₁ = sigmoid_cross_entropy( _{ID_GH} , False)

loss ₂ = sigmoid_cross_entropy( _{ID_H} ,True)

Among them, loss _D is the loss of the decision device, which calculates the loss and the corresponding gradient and sends it back to adjust the parameter model in the decision device.

S5. According to the above settings, train 200 epochs to achieve convergence, and save the model. The trained generator is used for subsequent denoising processing. The obtained image after denoising is _{ID_GH} , which is used as the input for subsequent image super-resolution. Next, define the satellite Image super-resolution model;

The super-resolution model also mainly includes three parts, namely the generator, the perceptual model and the decision device. The perceptual model and decision device use the same structure as that used in the previous denoising model.

Define the generator in the image super-resolution model: part of the main structure of the generator also uses a residual network, that is, build a residual module and then stack multiple residual modules to form the main body of the network structure, and then use sub The pixel convolution layer (subpixel), the specific structure is shown in Figure 5. The structure of the super-resolution generator is similar to the structure in the denoising model defined above. It adopts the mode of superposition of multiple residual modules, and the convolution layer uses 3* 3 convolution kernels, the number of convolution kernels is 64, the subsequent sub-pixel convolution layer and its correspondingly connected convolution layer use 256 convolution kernels, and the convolution layer uses 3*3 convolution kernels .In the super-resolution model that implements x2, the scale of the first sub-pixel convolutional layer=1, and the scale of the second sub-pixel convolutional layer=2.

The generator of the super-resolution model uses aerial photography data for training, and the input is an image pair composed of I _{SR_L} low-resolution 16-level aerial picture and its corresponding high-resolution 17-level aerial picture I _{SR_H} , and the output of the generator is I _{SR_GH} .

The loss function of the generator is defined as:

loss _{MSE_SR} ＝MSE(I _{SR_GH} ,I _{SR_H} )

S6. Repeat steps S3 to S5 to complete the super-resolution network training process and denoising model, and then generate a super-resolution image I _{SR_GH} . In order to further combine the clear priors in the aerial images into the satellite images, a Gaussian mixture model (GMM) is used here Build an external prior dictionary to guide the combination of internal image reconstruction and image sharpening to further improve the quality of the generated super-resolution satellite image;

S7. Constructing the GMM external prior dictionary to guide the internal image to reconstruct a clearer satellite image (originally used for image denoising). Here, the fact that the image pair cannot be formed between the aerial image and the satellite image cannot be directly used for training with the previously proposed generative confrontation network model. Using the method of constructing the GMM external prior dictionary can indirectly introduce the rich details in the clear aerial image to super-resolution In the generated satellite image; build the GMM external prior dictionary, divide the clear aerial photography 17-level image into small blocks of 15*15, and perform preliminary grouping (according to the Euclidean distance) after the block, as shown in Figure 6;

S701. Construct a GMM model according to the grouped image blocks, perform SVD decomposition on the covariance matrix in the obtained model, and construct a dictionary as an external prior to guide the reconstruction of subsequent satellite images;

S702. Using the I _{SR_GH} output in the previous super-resolution model as an internal image input, divide the input into blocks (15*15), and use the GMM model when constructing the external prior dictionary to guide the block for clustering;

S703. At the same time, use the dictionary formed by the external prior to guide the internal image blocks to construct the internal dictionary again;

S704. Perform sparse coding on the internal dictionary, and reconstruct a new internal graphic block group by combining the original internal image block group.

S8. Reconstruct the satellite image according to the grouping of the reconstructed internal graphics blocks, and perform an image sharpening operation on the reconstructed satellite image to make the edges in the image clearer to obtain a final result image.

The invention realizes the super-resolution of the satellite image under certain conditions by combining the multi-scale perceptual loss with the generation confrontation network. Among them, satellite images are used to train a network for denoising purposes, aerial images are used to train a network for image super-resolution, and the Gaussian mixture model is used to extract the feature prior of clear aerial images, and the super-resolution reconstructed The image is further reconstructed. After another Gaussian filter, the edges in the image are sharpened, and finally a clearer satellite image is produced.

The invention solves the problems of image resolution and image quality improvement under limited conditions. By using a multi-scale perceptual distortion loss, the multi-scale constraints on the feature domain of the generated image are realized to generate better images.

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. The components of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

A. Experimental conditions

1. The experiment uses the database

The data used in the experiment of the present invention are satellite image data and aerial image data provided in the satellite image super-resolution project. Non-public datasets, only partly shown here. Satellite imagery data contains:

Data type 1: Satellite images extracted from level 16 (including grainy and obvious noise), the clarity is not high;

Data type 2: Satellite images extracted from level 18 (no obvious granular noise), slightly higher definition. When the satellite images extracted from level 18 are down-sampled to level 16, satellite images that are clearer than satellite images extracted from level 16 can be obtained. However, due to the high cost of satellite images extracted from level 18, it is generally difficult to obtain a large number of them. Generally, satellite images extracted from level 16 are common. Therefore, the realization of this project is to train the image super-resolution model on the basis of obtaining a small part of the satellite images extracted from the 18th level, and then use the image super-resolution technology to obtain similar It even exceeds the extraction of satellite images from level 18 (assisted by using clear aerial images), which has great research significance and value. This type of data is less obtained here, but it overlaps with the coverage area in data type 1, so a small number of image pairs can be formed for model training.

Data type 3: Clear aerial data, due to the shooting height and shooting method, it is clearer than satellite images. Among aerial images of the same level as satellite images, aerial images are much clearer and contain rich texture information. However, the coverage of aerial images is limited, and the sources are limited. It is impossible to obtain aerial images of the same time period as the same location in data type 1 and data type 2. There is no image pair composed of satellite images and aerial images, which cannot be directly used for training, as shown in Table 1. shown.

Table 1 Datasets and their distribution

Data Type/Classification level 15 Level 16 17th grade 18th grade total data type 1 12989 51956 none none 64945 data type 2 1583 6332 25328 101302 134555 data type 3 1689 7104 27988 111952 148733

2. Experimental requirements

The experiment is divided into three parts: denoising model training, image super-resolution model training and image post-processing experiments.

Denoising model training: use the image pair composed of the satellite image extracted from level 16 (with noise) and the satellite image extracted from level 18 and downsampled to level 16 (no noise, but not high definition). As training data, train the generative adversarial network proposed in this scheme. After the training is complete, use the generator model to input a satellite image with noise to get a satellite image without noise. In order to ensure the robustness of the model, the test uses satellite images of different urban areas from the training, and is also a noisy image extracted from level 16.

Image super-resolution model training: Model training uses 17 levels of aerial images and 16 levels of aerial images obtained by downsampling of 17 levels to form image pairs for model training. After training the generative confrontation network proposed in this program, using the generator model, input a noise-free 16-level satellite image to generate a corresponding 17-level high-resolution image. and compare the visual effects of generating high-resolution images

Image post-processing experiment: For satellite images that have undergone denoising and image super-resolution processing, image post-processing is performed to further improve image quality. First, using the clear 17-level aerial image training to obtain the GMM external prior dictionary as a guide, input the 17-level satellite image obtained by super-resolution, construct the internal dictionary and reconstruct the image under the guidance of the external prior, and obtain a combination of clear and prior images in the aerial image. satellite imagery. On this basis, the Gaussian filtering method is used to sharpen the image to obtain the final post-processed image. Compare the clarity and visual quality of the resulting image to the original image.

3. Experimental parameter settings

The same settings are used to train the denoising model and the image super-resolution model. The first is the initialization training of the generator, the initial learning rate is set to 0.0001, and the training period is 100 epochs (one epoch is all training data). When the network is trained as a whole, the initial learning rate is still set to 0.0001, the training cycle is set to 200 epcoh, and the learning rate decays once when the training cycle reaches half, and decays to 0.00001.

In the image post-processing, constructing the external prior dictionary of the GMM model includes the following parameters: set the block step size to 3, the block size to 15*15, and select the 10 image blocks with the closest Euclidean distance as a group during clustering. The GMM model contains 32 Gaussian models, which fit 32 categories. Gaussian filtering is used for image sharpening, the filter radius is set to 1.5, and the sharpening strength is set to 2.

B. Experimental result evaluation criteria

Since the actual test input is the satellite image (including noise) extracted from level 16, there is no corresponding clear satellite image of level 17. It cannot be measured directly using common metrics such as PSNR and SSIM. Here, the effectiveness of this scheme is illustrated by comparing the graphs of some test results.

C. Comparative test plan

Please refer to Figure 7 and Figure 8, the test results listed above show the effectiveness of the proposed scheme in actual situations. Due to the restrictive conditions in the background of this solution, general image super-resolution algorithms cannot be directly trained and processed, and a series of image processing algorithms are needed to achieve the desired effect. Based on the original 16-level satellite image containing noise, the final test generated image effect not only removes the noise, but also achieves super-resolution to 17 levels (that is, length and width*2 in size). And with the help of clear aerial photography (not at the same location), the clarity of the generated 17-level satellite image has been improved and improved.

The above content is only to illustrate the technical ideas of the present invention, and cannot limit the protection scope of the present invention. Any changes made on the basis of the technical solutions according to the technical ideas proposed in the present invention shall fall within the scope of the claims of the present invention. within the scope of protection.

Claims (10)

1. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori, which is characterized in that contained using 16 grades Noise image corresponding 16 grades constitute image to training denoising model without noise image, then utilize data training of taking photo by plane Image Super-resolution model；Using priori dictionary outside Aerial Images building GMM model, and guide internal unsharp satellite image It is rebuild, completes the post-processing to the super resolution image of generation, then carry out image sharpening using the mode of gaussian filtering, most The full resolution pricture of protosatellite image is obtained eventually, realizes that the visual quality of images in protosatellite image basis is promoted.

2. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori according to claim 1, It is characterized in that, comprising the following steps:

S1, definition generate the generator in confrontation network, and network is lost in decision device and multiple dimensioned perception；

S2, using in existing satellite data since 18 grades extract image down sampling to 16 grades, if obtain 16 grades of satellite mappings As the target I as denoising_{D_H}, the satellite data extracted from 16 grades is as noisy image I_{D_L}Image pair is constituted, if generating not Noisy satellite image is I_{D_GH}；

S3, the image pair to be constituted in step S2, carry out initialization training to the generator in denoising model, train in initialization In, using mean square error as loss function, calculate the equal of the pixel between the corresponding target image of image that generator generates Square error obtains MSE generator loss function loss_MSE, calculate gradient and return adjustment model parameter；

S4, after the initialization of 100 epoch training, carry out complete model training, calculate loss and corresponding gradient simultaneously Network VGG19 not adjusting parameter is lost in parameter model in passback adjustment generator and decision device, perception；

S5, reach convergence, preservation model according to 200 epoch of arrangement above training, trained generator makes for denoising With the image after the denoising of acquisition is I_{D_GH}As the input of Image Super-resolution, satellite image super-resolution model is defined；

S6, step S3~S5 is repeated, completes super-resolution network training process and denoising model, then generates super resolution image I_{SR_GH}, external priori dictionary is constructed using gauss hybrid models；

S7, building GMM outside priori dictionary, 17 grades of images of clearly taking photo by plane are divided into the fritter of 15*15, then according to it is European away from From being tentatively grouped；

S8, the inside graph block packet reconstruction satellite mapping according to reconstruction carry out image sharpening operation to the satellite mapping of reconstruction, obtain Final result figure.

3. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori according to claim 2, It is characterized in that, in step S1, generates the generator in confrontation network is defined as: use a residual error network as generator, it is residual Include 16 residual error modules in poor network, includes three convolutional layers in each residual error module；

Decision device structure is defined as: use one 10 layers of convolutional neural networks as decision device, the convolution of convolutional neural networks Layer uses empty convolution；

Multiple dimensioned perception loss is defined as: use the VGG19 net that pre-training is crossed on IMAGENET1000 class taxonomy database Network perceptually loses network, by using conv2_2, conv3_4, conv4_4, Analysis On Multi-scale Features figure in multilayer, and building Multiple dimensioned perception loss.

4. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori according to claim 2, It is characterized in that, in step S3, MSE generator loss function loss_MSEIt is as follows:

loss_MSE=MSE (I_{D_GH},I_{D_H})。

5. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori according to claim 2, It is characterized in that, in step S4, when model training, by the MSE generator loss function loss in generator loss function_MSE, perception Loss function loss_vggWith confrontation loss function loss_GANGenerator loss function after weighting summation when the training of composition entirety is such as Under:

loss_G=loss_MSE+loss_vgg+loss_GAN。

6. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori according to claim 5, It is characterized in that, perception loss loss_vggIt is as follows:

loss_vgg=10^-6×(loss_{mse_conv2_2}+loss_{mse_conv3_4}+loss_{mse_conv4_4})

loss_{mse_conv2_2}=MSE (f_{i_conv2_2},f_{t_conv2_2})

loss_{mse_conv3_4}=MSE (f_{i_conv3_4},f_{t_conv3_4})

loss_{mse_conv4_4}=MSE (f_{i_conv4_4},f_{t_conv4_4})

Wherein, f_{i_conv2_2}, f_{i_conv3_4}, f_{i_conv4_4}Image is generated for input, and conv2_2, conv3_ are corresponded into sensor model 4, conv4_4 layers of characteristic pattern, f_{t_conv2_3}, f_{t_conv3_3}, f_{t_conv4_3}It is corresponded in target image input sensor model to generate image Obtained correspondence conv2_2, conv3_4, conv4_4 layer characteristic pattern；

Fight loss function loss_GANIt is as follows:

loss_GAN=10^-4×cross_entropy(I_{D_GH},True)

cross_entropy(I_{D_GH}, True) and=log (D (I_{D_GH}))

Wherein, D () is decision device.

7. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori according to claim 2, It is characterized in that, in step S4, decision device loss function loss when entirety is trained_DIs defined as:

loss_D=loss₁+loss₂

loss₁=sigmoid_cross_entropy (I_{D_GH},False)

loss₂=sigmoid_cross_entropy (I_{D_H},True)。

8. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori according to claim 2, Be characterized in that, in step S5, super-resolution model includes generator, sensor model and decision device, sensor model and decision device with go Structure used in model of making an uproar is identical, and the generator defined in Image Super-resolution model is as follows:

By building residual error module, then multiple residual error modules fold composition network structure main body, pass through the realization pair of sub-pix convolutional layer The amplification of image uses.

9. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori according to claim 8, It is characterized in that, the data that the generator training of super-resolution model uses are data of taking photo by plane, and are inputted as I_{SR_L}16 grades of boats of low explanation 17 grades of figure I that take photo by plane of bat figure high-resolution corresponding with its_{SR_H}The image pair of composition, generator output are I_{SR_GH}, define generator Loss function is as follows:

loss_{MSE_SR}=MSE (I_{SR_GH},I_{SR_H})。

10. a kind of satellite image ultra-resolution method for fighting network integration Aerial Images priori according to claim 2, It is characterized in that, step S7 is specific as follows:

S701, GMM model is constructed according to the image block of grouping, SVD decomposition, building is carried out to covariance matrix in obtained model Dictionary guides the reconstruction of satellite image below as external priori；

S702, the I to be exported in the super-resolution model of front_{SR_GH}It is inputted as internal image, 15*15 piecemeal is pressed after input, utilized GMM model guidance piecemeal when constructing external priori dictionary is clustered；

S703, the dictionary guidance internal image block constituted using external priori again pull up internal dictionary；

S704, the inside graph block grouping for carrying out sparse coding to internal dictionary, and combining former internal image block packet reconstruction new.