CN114331922B - Multi-scale self-calibration aero-optical effect turbulent degradation image restoration method and system - Google Patents

Abstract

The invention discloses a multi-scale self-calibration aero-optic effect turbulence degraded image restoration method, comprising the following steps: S1, extracting the feature map of the original aero-optic effect turbulence degraded image; S2, performing a pre-built self-calibration network on the feature map. Calibration to obtain a local fusion feature map calibrated for the local blurred region of the turbulent degraded image; S3. Perform multi-scale convolution recovery on the feature map of the original aero-optic effect turbulent degraded image to obtain a global restored feature map for the global region; S4. The local fusion feature map and the global restoration feature map are merged, and image restoration is performed on the merged feature map through convolution. The invention can take into account the multi-scale information of the image while utilizing the potential high and low resolution spatial information of the image, thereby accurately restoring the aero-optic effect turbulent degraded image.

Description

Multi-scale self-calibration aero-optical effect turbulent degradation image restoration method and system

technical field

The invention relates to the field of image processing, in particular to a method and system for restoring aero-optic effect turbulent flow degradation images based on a multi-scale self-calibration network.

Background technique

When the aircraft reaches a certain speed, the compression effect of the surrounding air will change the density of the flow field, which will adversely affect the imaging performance of the optical system. Image blur caused by aero-optical effects can be affected by various factors. Different from the degraded images in life, the process of turbulent image restoration has more complicated factors: 1. The change of the atmospheric refractive index will affect the resolution of the image during the imaging process of the optical system; 2. The low-level wind shear in the air layer and The instability in the air layer, the influence of the turbulent medium and the imperfect equipment during the transmission process lead to the degradation of the image quality; 3. The degradation of the original image is unknown, and it is difficult to estimate the degradation model from the turbulent degraded image, and various natural phenomena cause interference from random factors.

The existing traditional aero-optic effect degraded image restoration methods usually adopt three methods: 1. Flow field control method; 2. Adaptive optics method; 3. Digital image restoration method. The above methods can reduce the influence of aero-optical effects, but the point spread function solved by the existing algorithms will always have deviations.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide a method that can improve the restoration accuracy of optical effect turbulent degraded images.

The technical scheme adopted in the present invention is:

Provided is a multi-scale self-calibration aero-optic effect turbulent degradation image restoration method, comprising the following steps:

S1. Extract the feature map of the original aero-optic effect turbulent degraded image, the size of the feature map is C × H × W , where the channel dimension is C , and H and W are the height and width of the turbulent degraded image, respectively;

S2. The feature map is calibrated through a pre-built self-calibration network, and the feature map is divided into two sub-feature maps along the channel dimension. The number of channels of each sub-feature map is C/ 2, and the height of one of the sub-feature maps is extracted. , low-resolution spatial features, and perform weighted fusion to obtain calibration spatial features; extract the original resolution spatial features of another sub-feature map; fuse the original resolution spatial features and calibration spatial features to obtain local blurring for turbulent degraded images Locally fused feature maps for regional calibration;

S3. Perform multi-scale convolution recovery on the feature map of the original aero-optical effect turbulent degraded image to obtain a global recovery feature map for the global region;

S4. Combine the local fusion feature map and the global restoration feature map, and perform image restoration on the combined feature map through convolution.

Following the above technical solution, in step S2, the fusion feature map is used as the input, and the fusion operation is repeated m times, and the m fusion feature maps are cascaded to obtain the final fusion feature map, where m is a natural number.

Following the above technical solution, in step S2, the high-resolution and low-resolution spatial features are weighted and fused using the sigmoid function to obtain calibration spatial features.

Following the above technical solution, in step S2, a convolution layer is specifically used to extract the feature map of the turbulent degraded image.

In connection with the above technical solution, in step S3, a multi-channel filter is specifically used, and the hole rates of the convolution kernels in different channels are different, thereby forming a multi-scale convolution, and the number of channels of the multi-channel filter is equal to the channel dimension of the feature map. C;

Multi-scale convolution is used to expand the receptive field of the feature map of the original aero-optic effect turbulent degraded image, and the feature map after multi-scale convolution is output after the global region restoration.

The present invention also provides an aero-optical effect turbulent degradation image restoration system based on a multi-scale self-calibration network, comprising:

The feature image extraction module is used to extract the feature map of the original aero-optic effect turbulent degraded image. The size of the feature map is C × H × W , where the channel dimension is C , and H and W are the height and width of the turbulent degraded image, respectively;

The local fuzzy area calibration module is used to calibrate the feature map through a pre-built self-calibration network. Specifically, the feature map is separated into two sub-feature maps along the channel dimension, and the number of channels of each sub-feature map is C/ 2. The high and low resolution spatial features of a sub-feature map are weighted and fused to obtain calibration spatial features; the original resolution spatial features of another sub-feature map are extracted; the original resolution spatial features and calibration spatial features are fused to obtain Local fusion feature maps calibrated for local blurred regions of turbulent degraded images;

The global region restoration module is used to perform multi-scale convolution restoration on the feature map of the original aero-optical effect turbulent degraded image, and obtain the global restoration feature map for the global region;

The image restoration module is used to merge the local fusion feature map and the global restoration feature map, and perform image restoration on the merged feature map through convolution.

Following the above technical solution, the local fuzzy area calibration module is further used to specifically take the fusion feature map as an input, repeat the fusion operation m times, and cascade m fusion feature maps as the final fusion feature map, where m is a natural number.

Following the above technical solution, the local fuzzy area calibration module specifically uses the sigmoid function to perform weighted fusion on the high-resolution and low-resolution spatial features to obtain the calibrated spatial features.

Following the above technical solutions, the global area recovery module is specifically used for:

Specifically, a multi-channel filter is used, and the hole rates of the convolution kernels in different channels are different, thereby forming a multi-scale convolution, and the number of channels of the multi-channel filter is equal to the channel dimension C of the feature map;

Multi-scale convolution is used to expand the receptive field of the feature map of the original aero-optic effect turbulent degraded image, and the feature map after multi-scale convolution is output after the global region restoration.

The present invention also provides a computer storage device in which a computer program executable by a processor is stored, and the computer program executes the multi-scale self-calibration aero-optic effect turbulent degradation image restoration method described in the above technical solution.

The beneficial effects produced by the present invention are: the present invention obtains the calibration spatial features by weighted fusion of the high-resolution and low-resolution spatial features through the self-calibration network, and then fuses the calibration spatial features and the original resolution spatial features of the aero-optical effect turbulent degraded image feature map, The local area of the image is calibrated while retaining the original image features, and the local fusion feature map is obtained. After multiple calibrations, local blurred regions are recovered more accurately.

Further, the multi-scale convolution uses the dilated convolution of different dilation rates to expand the receptive field in the global blurred area to different degrees and integrates features of different scales, which can learn a larger blurring range and multi-scale information. Extract detailed information that is beneficial to the restored image. It can be seen that the present invention not only calibrates the local blur area in the image, but also effectively restores the details in the global area in the image, thereby improving the overall restoration quality of the image.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings and embodiments, in which:

1 is a flowchart of a method for restoring a multi-scale self-calibrating aero-optic effect turbulent degraded image according to an embodiment of the present invention;

2 is a flowchart of a method for restoring a multi-scale self-calibration aero-optic effect turbulent degraded image according to another embodiment of the present invention;

3 is a schematic diagram of a multi-scale self-calibration network according to an embodiment of the present invention;

Fig. 4 is the self-calibration process diagram of the local blurred area of the image according to the present invention;

Fig. 5 is the test result of the embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

As shown in FIG. 1 , the multi-scale self-calibration aero-optic effect turbulence degradation image restoration method according to an embodiment of the present invention includes the following steps:

S101. Extract the feature map of the original aero-optic effect turbulent degraded image, the size of the feature map is C × H × W , where the channel dimension is C , and H and W are the height and width of the turbulent degraded image, respectively;

S102. Calibrate the feature map through a pre-built self-calibration network, and specifically separate the feature map into two sub-feature maps along the channel dimension, the number of channels of each sub-feature map is C/ 2, and extract the height of one of the sub-feature maps. , low-resolution spatial features, and perform weighted fusion to obtain calibration spatial features; extract the original resolution spatial features of another sub-feature map; fuse the original resolution spatial features and calibration spatial features to obtain local blurring for turbulent degraded images Locally fused feature maps for regional calibration;

S103, performing multi-scale convolution recovery on the feature map of the original aero-optic effect turbulent degraded image, to obtain a global recovery feature map for the global region;

S104 , combine the local fusion feature map and the global restoration feature map, and perform image restoration on the combined feature map through convolution.

Further, in step S102, the fusion feature map may be used as an input, and the fusion operation may be repeated m times, and the m fusion feature maps may be cascaded to obtain the final fusion feature map, where m is a natural number.

In step S102, the high-resolution and low-resolution spatial features are weighted and fused using the sigmoid function to obtain calibration spatial features.

Further, in step S102, a convolution layer may be specifically used to extract the feature map of the turbulent flow degradation image.

In step S103, a multi-channel filter is specifically used, and the hole rates of the convolution kernels in different channels are different, thereby forming a multi-scale convolution, and the channel number of the multi-channel filter is equal to the channel dimension C of the feature map;

Multi-scale convolution is used to expand the receptive field of the feature map of the original aero-optic effect turbulent degraded image, and the feature map after multi-scale convolution is output after the global region restoration.

The present invention constructs multi-scale convolution through hole convolution with different hole rates. Different dilation rates have different degrees of expansion of the receptive field of the global blurred area, so multi-scale convolution helps to integrate the features of different scales (different receptive fields), which is helpful for blurred areas with different blur ranges and blur degrees. recovery.

Another embodiment of the present invention is a multi-scale self-calibration network-based aero-optical effect turbulent degradation image restoration method, which can be written in python language on the Linux operating system platform, as shown in FIG. 2 , and includes the following steps:

S201, using aero-optic effect turbulence degradation image simulation software to obtain aero-optic effect sequence turbulence degradation image database, dividing the database into training set and test set, and adding real scene aero-optic effect turbulence degradation images to the test sample set;

S202, train the pre-built self-calibration network through the sample data in the training set, and the specific training process includes:

Input the turbulent degraded image of aero-optical effect, extract the feature map of the turbulent-degraded image of aero-optical effect; pay attention to the local fuzzy area of the feature map of the turbulent-degraded image of aero-optical effect, and calibrate the local features of the feature map multiple times to obtain the calibration for the local fuzzy area of the turbulent degraded image The local fusion feature map of the aero-optics effect; learn the larger blur range and multi-scale information in the aero-optic effect turbulent degraded image feature map and restore the image details to obtain the global restoration feature map; merge the local fusion feature map and the global restoration feature map, and pass the volume The product restores the merged graph.

Extract the feature map of the original aero-optic effect turbulent degraded image, the size of the feature map is C × H × W , where the channel dimension is C , and H and W are the height and width of the turbulent degraded image, respectively;

The feature map is calibrated through a pre-built self-calibration network. Specifically, the feature map is divided into two sub-feature maps along the channel dimension. The number of channels in each sub-feature map is C/ 2, and the high and low values of one of the sub-feature maps are extracted. Resolution spatial features, and weighted fusion to obtain calibration spatial features; extract the original resolution spatial features of another sub-feature map; fuse the original resolution spatial features and calibration spatial features to obtain calibration for local blurred regions of turbulent degraded images The local fusion feature map of , takes the local fusion feature map as input, repeats the fusion operation m times, and then cascades m fusion feature maps as the final local fusion feature map, where m is a natural number.

Perform multi-scale convolution restoration on the feature map of the original aero-optic effect turbulent degraded image, and obtain the global restoration feature map for the global region;

Merge the local fusion feature map and the global restoration feature map;

Image restoration is performed on the merged feature maps by convolution.

The end-to-end training strategy can be used to continuously optimize the multi-scale self-calibration network through the training set until the optimal weight is obtained, and the trained self-calibration network is obtained;

S203, using the test sample set to test the trained self-calibration network, and evaluate the performance of the self-calibration network;

S204: Input the turbulent degraded image of the aero-optical effect to be restored into the self-calibration network that has been evaluated to meet the requirements, and output the restored image.

Further, as shown in FIG. 3 , in step S2, a feature extraction module is used to extract the feature map of the aero-optical effect turbulent degradation image. The module utilizes a single 3×3 convolutional layer to extract feature maps of size C × H × W turbulent degraded images of aero-optical effects. where C represents the number of feature map channels, and H and W represent the height and width of the feature map, respectively. The extracted aero-optic effect turbulent degradation image feature map is expressed as:

S _SF = C _SF ( I _SF )

I _SF represents the input aero-optic effect turbulent degraded image, C _SF (·) represents the convolution operation, and S _SF represents the obtained aero-optic effect turbulent degraded image feature map.

In step S3, according to the aero-optic effect turbulent degradation image feature map S _SF obtained in the previous step, it is divided into two sub-feature maps along the channel dimension C, and the channel number of each sub-feature map is C/2.

The self-calibration process of the present invention is shown in Figure 4, in which a sub-feature map is down-sampled and a convolutional layer is used for feature extraction to learn the representation of the low-resolution feature space. Simultaneously upsampling this sub-feature map and using convolutional layers for feature extraction to learn the representation of the high-resolution feature space. For an image I (dimension M×N), it is down-sampled by s times, that is, a resolution image of (M/s)×(N/s) size is obtained. Since the sub-feature map is down-sampling, it means that the sub-feature map is image reduced, and the resolution of this reduced image is low. When the convolution layer is used to extract features, what is learned is the representation of the low-resolution feature space. Similarly, for the up-sampled sub-feature map, it means that the sub-feature map is enlarged, so that a higher-resolution image can be obtained. When extracting features with a convolutional layer, what is learned is the high-resolution feature space. express.

The sigmoid function can be used for weighted fusion of high-resolution and low-resolution spatial features to obtain calibrated spatial features.

For another sub-feature map, a convolutional layer is used to extract features in the original resolution space.

The features of the original resolution space and the calibration features are fused to obtain the output Y ¹ .

The output Y ¹ obtained by fusion is used as the input, and the operation of step S3 is repeated to obtain the output Y ² , and so on, and the operation is repeated m times.

The m outputs ( Y ¹ ), ( Y ² ), …, ( Y ^m ) are cascaded, as shown in Figure 3, the cascade process of m outputs to obtain the total output Y ^M is shown, and Y ^M is expressed as:

Y ^M = C _sum [( Y ¹ ), ( Y ² ), …,( Y ^m )]

Y ^M represents the total output of the cascade ( Y ¹ ), ( Y ² ), …, ( Y ^m ), and C _sum represents the cascade operation on ( Y ¹ ), ( Y ² ), …, ( Y ^m ). Y ^m represents the mth output.

Further, as shown in FIG. 3, in step S4, a multi-channel filter is specifically used, and the hole rates of the convolution kernels in different channels are different, thereby forming a multi-scale convolution, and the number of channels of the multi-channel filter is equal to the feature map. The channel dimension C of ;

Multi-scale convolution is used to expand the receptive field of the feature map of the original aero-optic effect turbulent degraded image, and the feature map after multi-scale convolution is output after the global region restoration.

Further, in step S5, the concatenated output Y ^M of the aero-optical effect turbulent degraded image feature maps S _SF and SF is added pixel by pixel, so that the feature map after the calibration of the local fuzzy area and the feature after the restoration of the global fuzzy area are The graphs are combined and output S _DF :

S _DF= S _SF + W _SC C _sum [( Y ¹ ), ( Y ² ), …,( Y ^m )]= S _SF + W _SC Y ^M

W _SC is the weight of the 3 × 3 convolution set after Y ^M , and S _SF represents the original aero-optic effect turbulence degraded image feature map.

A 3×3 convolutional layer is used to reconstruct the combined feature map into a clear image: I _CI =C _Rec ( S _DF ), C _Rec represents a convolution operation, and I _CI represents the restored image.

，N表示数据集的总数量，H _DRN 表示整个多尺度自校准网络， I ⁱ _BI 表示训练集中的第i张气动光学效应湍流退化图像，I ⁱ _CI 表示第i张原图像。利用损失函数L(θ)优化多尺度自校准网络，更新网络参数θ，获得训练样本集的权重；Further, step S6 specifically adopts the training sample set, and constructs a loss function through automatic differentiation technology, using algorithms based on stochastic gradient descent and back propagation

, N represents the total number of datasets, H _DRN represents the entire multi-scale self-calibration network, I ⁱ _BI represents the ith aero-optic effect turbulent degradation image in the training set, and I ⁱ _CI represents the ith original image. Use the loss function L ( θ ) to optimize the multi-scale self-calibration network, update the network parameters θ , and obtain the weight of the training sample set;

On the basis of using the weights of the training set, the test sample set is used to test the multi-scale self-calibration network. The test uses a trained self-calibration network. The test is equivalent to sending the turbulent degraded image to the network, and it will directly come out the restored image. Intuitively it becomes clearer (visual effects), but generally post-test results are evaluated to verify the effectiveness of the network (data illustration). In the embodiment of the present invention, a peak signal-to-noise ratio (Peak signal-to-noise ratio, PSNR) evaluation index is used to evaluate the results after the test. This evaluation index is a common index for image quality evaluation, and the larger the value, the better the image quality.

The present invention provides an implementation case, using aero-optic effect turbulence degradation image simulation software to obtain aero-optic effect sequence turbulence degradation image database, dividing the database into training set and test set, and adding real scene aero-optic effect turbulence degradation to the test sample set Images; 1000 original images and 1000 aero-optic effect turbulent degradation images are used as training set, and 50 images are used as test data set. The images in the dataset are all 256*256 pixels in size.

The test results shown in Figure 5 show that the quality is better than the original input turbulent degraded images, which indicates that the trained network model is effective for the restoration of turbulent degraded images.

When testing the self-calibration network with the test sample, a peak signal-to-noise ratio (PSNR) evaluation index can be used to evaluate the reconstructed result. During each iteration of training, the initial learning rate is set to 10 ^-4 , the parameters are set to: m = M = 20, the overall model architecture is implemented on GeForce GTX Titan V with PyTorch, the training period is 800 epoch, and the training is completed and saved optimal model parameters.

In the testing phase, another 50 images were selected from the aero-optic effect turbulent degradation image data set as the test data set, in which the average PSNR of the degradation map=20.006dB. The optimal model parameters obtained by training are used for testing, and the restored image is obtained. The average time is 1.18s, and the average PSNR of the restored image is 32.361 dB. The test results are shown in Figure 5. It can be seen intuitively from the figure that what we input is a turbulent degraded image with PSNR=20.006dB. Based on the optimal model, the restored image is obtained after the entire network is restored, and the average PSNR of the restored image is 32.361dB. rises, which proves that our network is effective for the recovery of turbulent degraded images of aero-optical effects.

The present application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (eg, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic Disk, Optical Disc, Server, App Store, etc., on which computer programs, programs are stored The corresponding functions are implemented when executed by the processor. The computer-readable storage medium of this embodiment is used to implement, when executed by a processor, the multi-scale self-calibration aero-optic effect turbulent degradation image restoration method of the method embodiment.

It should be understood that, for those skilled in the art, improvements or changes can be made according to the above description, and all these improvements and changes should fall within the protection scope of the appended claims of the present invention.

Claims (10)

1. A multi-scale self-calibration aero-optical effect turbulence degradation image restoration method is characterized by comprising the following steps:

s1, extracting a characteristic diagram of the original aero-optical effect turbulence degradation image, wherein the size of the characteristic diagram isC×H×WWherein the channel dimension isC，H、WRespectively the height and width of the turbulence degradation image;

s2, calibrating the feature map through a pre-constructed self-calibration network, specifically, separating the feature map into two sub-feature maps along the channel dimension, wherein the number of channels of each sub-feature map isC/2, extracting high-resolution and low-resolution spatial features of one of the sub-feature maps, and performing weighted fusion to obtain calibration spatial features; extracting the original resolution spatial features of the other sub-feature map; fusing the original resolution spatial features and the calibration spatial features to obtain a local fusion feature map calibrated for a local fuzzy region of the turbulence degradation image;

s3, carrying out multi-scale convolution recovery on the characteristic diagram of the original aero-optical effect turbulence degradation image to obtain a global recovery characteristic diagram for a global area;

and S4, merging the local fusion feature map and the global restoration feature map, and restoring the image of the merged feature map by convolution.

2. The multi-scale self-calibration aero-optical effect turbulence degradation image restoration method according to claim 1, wherein in step S2, the local fusion feature map is used as an input, the fusion operation is repeated m times, and m fusion feature maps are cascaded to be used as a final local fusion feature map, wherein m is a natural number.

3. The multi-scale self-calibration aero-optical effect turbulence degradation image restoration method according to claim 1, wherein in step S2, the sigmoid function is used to perform weighted fusion on the high-resolution spatial features and the low-resolution spatial features to obtain calibrated spatial features.

4. The method for recovering the turbulence degradation image based on the multi-scale self-calibration aero-optical effect of claim 1, wherein in step S2, the convolution layer is specifically used to extract the feature map of the turbulence degradation image.

5. The method for restoring a turbulence-degraded image with a multi-scale self-calibration aero-optical effect according to claim 1, wherein in step S3, a multi-channel filter is used, and the void rates of convolution kernels in different channels are different, so as to form a multi-scale convolution, wherein the number of channels of the multi-channel filter is equal to the channel dimension C of the feature map;

and expanding the receptive field of the characteristic diagram of the original aerooptical effect turbulence degradation image by utilizing multi-scale convolution, and outputting the characteristic diagram which is recovered in the global area after the multi-scale convolution.

6. A pneumatic optical effect turbulence degradation image restoration system based on a multi-scale self-calibration network is characterized by comprising:

the characteristic image extraction module is used for extracting a characteristic diagram of the original aerodynamic optical effect turbulence degradation image, and the size of the characteristic diagram isC×H×WWherein the channel dimension isC，H、WRespectively the height and width of the turbulence degradation image;

a local fuzzy region calibration module, configured to calibrate the feature map through a pre-constructed self-calibration network, specifically, separate the feature map into two sub-feature maps along a channel dimension, where the number of channels in each sub-feature map isCExtracting high-resolution and low-resolution spatial features of one of the sub-feature maps, and performing weighted fusion to obtain calibration spatial features; extracting the original resolution spatial features of the other sub-feature map; fusing the original resolution spatial features and the calibration spatial features to obtain a local fusion feature map calibrated for a local fuzzy region of the turbulence degradation image;

the global region recovery module is used for performing multi-scale convolution recovery on the feature map of the original aero-optical effect turbulence degradation image to obtain a global recovery feature map for a global region;

and the image restoration module is used for merging the local fusion feature map and the global restoration feature map and restoring the image of the merged feature map by convolution.

7. The system for restoring the turbulence degradation image of the aero-optical effect based on the multi-scale self-calibration network according to claim 6, wherein the local fuzzy region calibration module is further configured to repeat the fusion operation m times by using the fusion feature map as an input, and cascade m fusion feature maps as a final fusion feature map, wherein m is a natural number.

8. The pneumatic optical effect turbulence degradation image restoration system based on the multi-scale self-calibration network according to claim 6, wherein the local fuzzy region calibration module performs weighted fusion on the high-resolution spatial features and the low-resolution spatial features by using a sigmoid function to obtain calibrated spatial features.

9. The system for restoration of an aero-optical effect turbulence degradation image based on multi-scale self-calibration network according to claim 6, wherein the global area recovery module is specifically configured to:

specifically, a multi-channel filter is used, the void rates of convolution kernels in different channels are different, so that multi-scale convolution is formed, and the number of channels of the multi-channel filter is equal to the channel dimension C of the characteristic diagram;

and expanding the receptive field of the characteristic diagram of the original aerooptical effect turbulence degradation image by utilizing multi-scale convolution, and outputting the characteristic diagram which is recovered in the global area after the multi-scale convolution.

10. A computer storage device having stored therein a computer program executable by a processor to perform the multi-scale self-calibrating aero-optical effect turbulence-degraded image restoration method of any one of claims 1-5.

Images