CN112767252A - Image super-resolution reconstruction method based on convolutional neural network - Google Patents

Abstract

The invention belongs to the field of image technology, in particular to an image super-resolution reconstruction method based on a convolutional neural network. According to the difference principle of receptive field of different convolution kernels, the method of the invention improves the convolutional neural network by using special-shaped convolution kernels, so that the network can extract the rich features of the image under the condition of less parameters. At the same time, the multi-layer feature image information is fully utilized by using the tanh function as the activation function and the skip connection, which improves the quality of the reconstructed image. In addition, the method of the present invention utilizes the light-weight principle of the convolutional neural network to optimize the weight of the convolutional neural network, so that the parameters of the entire network are reduced and the computation amount is reduced when the reconstructed image quality of the network is slightly reduced. The method of the present invention can improve the quality of the reconstructed image on the basis that the parameters of the traditional network are not greatly improved, and at the same time, the reconstructed image has an obvious improvement in the human eye perception compared with the image reconstructed by the traditional method.

Description

An image super-resolution reconstruction method based on convolutional neural network

technical field

The invention belongs to the technical field of image processing, in particular to an image super-resolution reconstruction method based on a convolutional neural network.

Background technique

Image super-resolution reconstruction is an image processing technique that reconstructs low-resolution images into high-resolution images. With the improvement of device display capabilities, users need higher resolution images to bring better visual experience, and higher resolutions can also facilitate scientific research by practitioners in more industries. Resolution reconstruction.

Image super-resolution reconstruction methods include interpolation-based image super-resolution reconstruction, fitting-based image super-resolution reconstruction and learning-based image super-resolution reconstruction. Image super-resolution reconstruction based on learning is a popular reconstruction method at present. It has the characteristics of good reconstruction quality, rich reconstructed image details and good visual perception. It has been widely used in medical, satellite and civil fields. Therefore, It is of great significance to improve the reconstruction quality of image super-resolution reconstruction. The image super-resolution reconstruction method based on learning includes the image super-resolution reconstruction method based on dictionary learning and the image super-resolution reconstruction method based on convolutional neural network. The image super-resolution reconstruction method based on convolutional neural network has The better feature extraction ability and nonlinear mapping ability make the reconstructed image quality better than other methods.

At present, the image super-resolution reconstruction methods based on convolutional neural networks mainly focus on the following:

1. Convolutional Neural Networks

The convolutional neural network method uses the convolutional neural network in the problem of image super-resolution reconstruction. This method mainly uses three layers of convolutional layers to perform super-resolution reconstruction of images: feature extraction layer, nonlinear mapping layer and super-resolution. Reconstruction layer, this method effectively improves the quality of the reconstructed image, but due to the disadvantages of few network layers and low utilization of image features, the image information is not fully used, and the reconstruction effect can be further improved.

2. Recurrent Neural Network

The cyclic neural network method is to use the cyclic neural network to perform super-resolution reconstruction of the image. After the feature extraction layer, the principle of cyclic convolution is used to perform cyclic convolution on the feature image, and the result of each convolution is passed to the last one. Layer convolution layer to realize the extraction of image features. Although the method of image super-resolution reconstruction using cyclic neural network has not changed much, the cyclic convolution method can fully extract some features of the image, avoiding some convolutional neural networks with few layers and insufficient feature extraction. However, the cyclic neural network uses the same convolution layer for cyclic convolution, which cannot fully extract different features of the image, so that the reconstructed image will have a better reconstruction effect on certain types of recurring features, and other features have a general effect.

3. Adversarial Neural Networks

This method is based on the adversarial neural network, and uses the generative network and the adversarial network to train at the same time, and obtains a pattern with rich texture that can be reconstructed. The content presented by the reconstructed image is quite different from the real image, and training the two networks at the same time requires a long time and more resources, which requires very high equipment and has certain limitations in practicability.

SUMMARY OF THE INVENTION

The purpose of the present invention is to propose an image super-resolution reconstruction method based on an improved convolutional neural network-Horizonal and Vertical Super-Resolution (HVSR) network, in view of the above-mentioned problems, which is a kind of reconstructed image with good performance. And image super-resolution reconstruction method with fewer network parameters.

The technical solution of the present invention is, a method for image super-resolution reconstruction based on convolutional neural network, as shown in FIG. 1 , comprising the following steps:

S1. Select n images as the training set {P _H1 , P _H2 ,..., P _Hn }, and the subscript H indicates that the images are high-resolution images;

S2. Preprocess the training set: randomly extract a pixel area with a size of 100 × 100 in each image of the training set. If the pixel area of the image is less than 100, fill in the insufficient area with 0, and then downsample these pixel areas to obtain the corresponding The set of low-resolution images {P _L1 ,P _L2 ,...,P _Ln };

S3. Construct a convolutional neural network, and define the direction of the network from input to output as the vertical direction, then:

Along the vertical direction, it includes the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer, the sixth convolutional layer, the deconvolutional layer, and the seventh convolutional layer. layered; of which,

The input of the first convolution layer is a low-resolution image set, and the feature images output by the first convolution layer are divided into two groups of equal numbers;

The second convolution layer has two convolution kernels of a×1 and 1×a. The two convolution kernels take the two sets of outputs of the first convolution layer as inputs, and the output of each convolution kernel is divided into two groups. For two groups of equal numbers, that is, the second convolutional layer outputs four groups of feature images;

The third convolution layer has four convolution kernels, which are alternately composed of two convolution kernels of a×1 and 1×a along the horizontal direction. The first group of a×1 and 1×a convolution kernels correspond to The input is the two sets of feature images output by the a×1 convolution kernel in the second convolution layer, and the input corresponding to the second group of a×1 and 1×a convolution kernels is the 1×a convolution kernel in the second convolution layer. The output of two sets of feature images, the output of each convolution kernel is divided into two groups of equal numbers, that is, the third convolution layer outputs eight sets of feature images;

The fourth convolution layer has eight convolution kernels, which are alternately composed of two convolution kernels of a×1 and 1×a along the horizontal direction. The input of each convolution kernel is the same as that of the third convolution layer. In the same way as the product kernel, each convolution kernel outputs a set of feature images;

The fifth convolution layer is a 1×1 convolution kernel, the input is all feature images output by the second convolution layer, the third convolution layer, and the fourth convolution layer, and the number of output feature images is the first layer feature half the number of images;

After each convolution in the network, the tanh function is used as the activation function;

将均方差作为网络的损失函数，对网络参数进行更新，从而获得训练好的卷积神经网络；S4. Train the constructed convolutional neural network, specifically: setting the target image as a residual image, and enlarging the low-resolution image set to the same size as the corresponding image in the training set by using the bilinear interpolation method, so as to obtain an enlarged low-resolution image. Resolution image set {P _IL1 ,P _IL2 ,...,P _ILn }, the pixel value of the corresponding position of the corresponding image in the extracted pixel area is subtracted from the enlarged low-resolution image set to obtain the residual image set {P _R1 ,P _R2 ,...,P _Rn }; compare the output image P _Oi obtained after inputting the low-resolution image P _Li into the network with the corresponding residual image P _Ri to obtain the mean square error

The mean square error is used as the loss function of the network, and the network parameters are updated to obtain a trained convolutional neural network;

S5. Use the trained convolutional neural network to perform super-resolution reconstruction of the image, as shown in Figure 2, specifically:

S51, taking the image _PL that needs super-resolution reconstruction as the input image, and inputting the convolutional neural network to obtain the output image _PO ;

S52, utilize the bicubic interpolation method to enlarge the image _PL to be as large as the image P _O , and obtain the enlarged image P _IL ;

S53. Add the pixel values of the pixel points corresponding to P _IL and PO to obtain a super-resolution image _{P S .}

Further, the first convolution layer is a 3×3 convolution kernel, the second convolution layer is a 1×5 and 5×1 convolution kernel, and the sixth convolution layer is a 1×1 convolution kernel. kernel, the seventh convolution layer is a 3×3 convolution kernel.

The beneficial effects of the present invention are:

(1) By using 1×5 and 5×1 convolution kernels that are different from the traditional n×n convolution kernels, the receptive field is richer; at the same time, compared with the commonly used 3×3 convolution kernels, each convolution kernel The kernel has fewer parameters, but the same receptive field can be obtained in the end.

(2) Convolution kernels of 1×5 and 5×1 are used in one convolutional layer at the same time. While ensuring the extraction of horizontal and vertical features, the combination of different convolutional kernels in multi-layer convolutional layers Obtaining different receptive field shapes enriches the ability of convolutional neural networks to extract different shape features in images.

(3) The present invention adopts the idea of light weight, utilizes the idea of grouped convolution, and reduces the computation amount of the network on the basis that the reconstruction performance does not change greatly, so that the convolutional neural network has a wider application range.

Description of drawings

Fig. 1 is the training flow chart of the method of the present invention.

FIG. 2 is a flowchart of image super-resolution reconstruction according to the method of the present invention.

FIG. 3 is a structural diagram of a convolutional neural network of the present invention.

Figure 4 shows the performance of different models on the test set after different training cycles.

Figure 5 shows the super-resolution reconstruction effect of different models on the same image.

Detailed ways

Below in conjunction with accompanying drawing, the technical scheme of the present invention is described in detail:

A convolutional neural network is defined as:

Y=a(w*X+b)

Y represents the output, a( ) represents the activation function, w represents the convolution kernel, * represents the convolution operation, and b represents the bias term.

As shown in Figure 3, compared with the use of n×n convolution kernels for image super-resolution reconstruction when the traditional convolutional neural network performs super-resolution reconstruction, the present invention defines special-shaped convolution kernels using 1×5 and 5× The two convolution kernels of 1 replace the original convolution kernel. This convolution kernel can use fewer parameters to achieve the same receptive field effect as the traditional convolution kernel, such as using a 5×5 convolution kernel as a convolutional neural network. The size of the receptive field is 5 × 5, and the number of parameters used is 25. When a 3×3 convolution kernel is used as the convolution kernel of a convolutional neural network, two 3×3 convolution kernels of the same size are required to achieve a 5×5 receptive field, and 18 parameters are used at this time. When using two convolution kernels of 1 × 5 and 5 × 1 for convolution, a 1 × 5 and a 5 × 1 convolution kernel are required to achieve a 5 × 5 receptive field, and the parameters used at this time are 10 , which is much smaller than the number of traditional convolution kernels.

When the traditional convolutional neural network performs super-resolution reconstruction, each convolutional layer uses only one shape convolution kernel for convolution. After multi-layer convolution, a certain point in the feature image has a single receptive field shape. , the region of the extracted features of the original image is relatively fixed, and the method of the present invention places two convolution kernels of 1×5 and 5×1 in the same convolution layer, which can ensure the extraction of horizontal and vertical in the previous layer. The image information in the straight direction avoids the problem of not extracting image feature information in the other direction caused by using only a single direction to extract image information. At the same time, the combination of multiple convolution kernels can make the shape of the receptive field of the entire convolutional neural network. Rich, it is more beneficial to extract image features of certain shapes; after multi-layer convolution layers, the receptive fields of different shapes are combined with each other to obtain receptive fields of different shapes, and the receptive fields of different shapes have different shapes for image features of different shapes. With different extraction capabilities, compared with the receptive field of a single shape, the receptive field of different shapes can better capture different features of the image.

The network of the present invention adopts the method of grouping convolution to group the feature images extracted from each layer, and each group is convolved with the convolution kernel of the next layer. For example, the image extracted by the second layer convolution kernel is divided into There are two groups, each of which is convolved with 1×5 and 5×1 convolution kernels, respectively. In this way, the number of convolution kernels in each layer is unchanged, but by grouping, the amount of computation required for each layer is greatly reduced. For example, after the feature images extracted by the second layer are divided into two groups, the amount of computation required by the third convolution layer is only 50% of that of the second layer, and the feature images extracted by the third layer are divided into four groups. 25% of the second floor, and so on. And before the subsequent convolution, use the skip connection method to collect all the feature images, and use a 1×1 convolution kernel to "mix" all the feature images, and change the number of feature images to reduce the convolution of the latter layer. Layer operations.

After each convolution, the tanh function is used as the activation function, because in the image super-resolution reconstruction problem, the required output result is image information, the objects in the image are relatively smooth and continuous, and the processed target image has The value range is between [-1, 1], and the expression of the tanh function is as follows:

The output value of the tanh function is also between [-1, 1], unlike the ReLU and PReLU functions, which have non-derivative points at the point where the value is 0. The tanh function will not directly set the negative output value to 0, so that the image feature will not be activated, resulting in the loss of part of the image information during the training process. The output value of the tanh function near 0 has a large rate of change. Small changes can be better captured. The output value far from 0 has a small rate of change, and the output range is limited. For large convolution results, its growth can be well suppressed to prevent gradient disappearance and gradient. Explosion problem. Therefore, the feature image after passing through the activation function is closer to the target image, which is more conducive to the subsequent processing of image super-resolution reconstruction.

In training, the dropout method is used for training. After a certain number of training cycles, a part of the convolution kernels are randomly shielded for retraining. After training for a period of time, new convolution kernels are re-selected for shielding, and finally all the convolution kernels are combined. Prevent the entire convolutional neural network from overfitting and improve the robustness of the network.

The convolutional neural network of the present invention carries out lightweight processing, which is embodied as follows:

When the traditional convolutional neural network performs image super-resolution reconstruction, the method used is that the convolution kernel of the lower layer convolves all the feature images of the upper layer, so the traditional convolutional neural network is used in each convolutional layer. The amount of calculation required is large, and the required parameters are also more. The calculation amount of standard convolution is as follows:

Q=HWNKLM

H×W represents the spatial size of the input image (H represents the height of the image, W represents the width of the image), N represents the number of channels of the input image, and K×L represents the size of the convolution kernel (K represents the length of the convolution kernel, L represents the width of the convolution kernel), and M represents the number of convolution kernels (the number of channels of the output image).

In the present invention, the method of grouping convolution is adopted to group the feature images of the previous layer, and the convolution kernel of the latter layer only performs convolution on one of the grouped feature images, and the calculation amount of the convolution is for:

G is the number of groups into which the feature image is divided. It can be seen that the more the number of components, the smaller the amount of convolution operation, but at the same time, the previous layer extracted from each feature image in the feature image of the latter layer The smaller the number of feature images, when G takes the maximum value N, the entire network becomes a single-channel convolutional neural network.

As shown in Figure 3, the method of the present invention performs grouping processing on the extracted feature images in the 2-4 layers of the convolutional neural network, wherein the second layer is divided into two groups, which are divided into the features extracted by the 1×5 convolution kernel. The image group and the feature image group extracted by the 5×1 convolution kernel, and the convolution layer of the latter layer corresponding to each group of feature images contains two types of convolution kernels, 1×5 and 5×1, so that The purpose is to ensure the richness of the receptive field, so that the network can extract rich image features. The third layer divides the feature images into 4 groups according to 4 groups of convolution kernels, and so on. In the fourth layer of 8 groups, using the convolutional neural network after grouping convolution reduces 24% of the unused parameters, which greatly saves calculated amount.

At the same time, after using grouped convolution, the present invention combines the feature images in the first several layers together by means of skip connection, and uses a 1×1 convolution kernel to mix these images, so that the feature images of the latter layer are mixed together. The number is reduced, reducing the amount of operation and the number of network parameters of the convolution operation of the latter layer.

The effectiveness of the present invention is proved below through simulation example and in conjunction with Fig. 4, Fig. 5:

During the training process, 10,000 images in the ImageNet2012 dataset were used as the training set, and each image was randomly selected as the training area with a pixel size of 100 × 100, and the image magnification was set to 4 times. The learning rate is set to 0.0001.

Figure 4 is a comparison of the reconstruction performance of different network models in the test set in different training cycles. The abscissa is the number of cycles for which the model is trained, and the ordinate is the peak signal-to-noise ratio, which is an indicator for objectively evaluating the quality of image reconstruction. It can be seen from the figure that the network of the present invention obtains good image reconstruction quality after more than a dozen training cycles, which is superior to other networks. This example shows that the present invention has good feature extraction capability and image super-resolution reconstruction capability, and is easy to train.

Fig. 5 is the reconstruction effect of different network models. It can be seen from the figure that the image reconstructed by the network of the present invention has a certain improvement in the human eye perception compared with other networks, and the degree of blurring is lower. The network also has no jaggedness during reconstruction. This example shows that the image reconstructed by the present invention has a very good appearance to the human eye.

Claims (2)

1. an image super-resolution reconstruction method based on convolutional neural network, is characterized in that, comprises the following steps: S1. Select n images as the training set {P _H1 , P _H2 ,..., P _Hn }, and the subscript H indicates that the images are high-resolution images; S2. Preprocess the training set: randomly extract a pixel area with a size of 100 × 100 in each image of the training set. If the pixel area of the image is less than 100, fill in the insufficient area with 0, and then downsample these pixel areas to obtain the corresponding The set of low-resolution images {P _L1 ,P _L2 ,...,P _Ln }; S3. Construct a convolutional neural network, and define the direction of the network from input to output as the vertical direction, then: Along the vertical direction, it includes the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer, the sixth convolutional layer, the seventh convolutional layer, and the reverse convolutional layer. layered; of which, The input of the first convolution layer is a low-resolution image set, and the feature images output by the first convolution layer are divided into two groups of equal numbers; The second convolution layer has two convolution kernels of a×1 and 1×a. The two convolution kernels take the two sets of outputs of the first convolution layer as inputs, and the output of each convolution kernel is divided into two groups. For two groups of equal numbers, that is, the second convolutional layer outputs four groups of feature images; The third convolution layer has four convolution kernels, which are alternately composed of two convolution kernels of a×1 and 1×a along the horizontal direction. The first group of a×1 and 1×a convolution kernels correspond to The input is the two sets of feature images output by the a×1 convolution kernel in the second convolution layer, and the input corresponding to the second group of a×1 and 1×a convolution kernels is the 1×a convolution kernel in the second convolution layer. The output of two sets of feature images, the output of each convolution kernel is divided into two groups of equal numbers, that is, the third convolution layer outputs eight sets of feature images; The fourth convolution layer has eight convolution kernels, which are alternately composed of two convolution kernels of a×1 and 1×a along the horizontal direction. The input of each convolution kernel is the same as that of the third convolution layer. In the same way as the product kernel, each convolution kernel outputs a set of feature images; The fifth convolution layer is a 1×1 convolution kernel, and the input is all feature images output by the second convolution layer, the third convolution layer, and the fourth convolution layer; After each convolution in the network, the tanh function is used as the activation function; S4. Train the constructed convolutional neural network, specifically: setting the target image as a residual image, and enlarging the low-resolution image set to the same size as the corresponding image in the training set by using the bilinear interpolation method, so as to obtain an enlarged low-resolution image. Resolution image set {P _IL1 ,P _IL2 ,...,P _ILn }, the pixel value of the corresponding position of the corresponding image in the extracted pixel area is subtracted from the enlarged low-resolution image set to obtain the residual image set {P _R1 ,P _R2 ,...,P _Rn }; compare the output image P _Oi obtained after inputting the low-resolution image P _Li into the network with the corresponding residual image P _Ri to obtain the mean square error

The mean square error is used as the loss function of the network, the network parameters are updated, and the trained convolutional neural network is obtained after training with a preset number of training cycles; S5. Use the trained convolutional neural network to perform super-resolution reconstruction of the image, specifically: S51, taking the image _PL that needs super-resolution reconstruction as the input image, and inputting the convolutional neural network to obtain the output image _PO ; S52, utilize the bicubic interpolation method to enlarge the image _PL to be as large as the image P _O , and obtain the enlarged image P _IL ; S53. Add the pixel values of the pixel points corresponding to P _IL and PO to obtain a super-resolution image _{P S .} 2. A method for image super-resolution reconstruction based on convolutional neural network according to claim 1, wherein the first convolutional layer is a 3×3 convolution kernel, and the second convolutional layer is are 1×5 and 5×1 convolution kernels, the sixth convolution layer is a 1×1 convolution kernel, and the seventh convolution layer is a 3×3 convolution kernel.

Images