CN114782298B - Infrared and visible light image fusion method with regional attention - Google Patents

Abstract

Infrared and visible light image fusion aims to take advantage of information complementarity to fuse thermal radiation, texture details and other information in the same scene, making the fused image content more comprehensive and clear, and conducive to human eye observation and subsequent tasks. The steps of image fusion are usually feature extraction, feature fusion and image reconstruction. The present invention proposes a fusion method with regional attention. First, the encoder is used to extract high-dimensional features, then a fusion strategy with significant regional attention is designed to fuse the features, and finally the decoder is used to reconstruct the image. The present invention aims to solve the problem of image fusion in scenes with insufficient lighting. The results show that the present invention can fully retain the good texture details of visible light images, and use infrared images to supplement the content of underexposed areas. In addition, the present invention focuses on salient areas so that the highlighted areas in the source image remain highlighted in the fused image, achieving a good effect of complementing the advantages of infrared and visible light images.

Description

An infrared and visible light image fusion method with regional attention

Technical field

The invention belongs to the technical field of image processing, and particularly relates to an infrared and visible light image fusion method with regional attention.

Background technique

With the steady development of the hardware and software industries, the ability to use sensors to collect information, and to transmit and process information is also increasing. In this context, vision-based sensors have been widely used because they can provide rich environmental information. A single type of sensor only has information characteristics that represent a certain aspect and cannot fully describe the monitoring environment. Therefore, multi-sensor systems have begun to receive more and more attention and applications. The multi-source sensor imaging system completely fills the gap of the insufficient image expression ability of a single sensor. At present, image fusion technology has played a significant application value in remote sensing detection, safety navigation, medical image analysis, anti-terrorism inspection, environmental protection, traffic monitoring, clear image reconstruction, disaster detection and forecasting, especially in computer vision and other fields.

For visual multi-source sensor systems, infrared and visible light images can be obtained through relatively simple equipment. The most typical one is the fusion of infrared and visible light images. Due to the different imaging mechanisms of the two, visible images usually have high spatial resolution and image contrast, which are suitable for human visual perception. However, they are highly susceptible to harsh conditions, such as insufficient brightness, heavy rain, haze and other special climates. However, infrared images happen to have better scene anti-interference capabilities, and targets with temperatures higher than the environment, such as pedestrians, can be displayed more significantly. However, infrared images usually have low resolution and poor image details. Fusing the two can display multiple information on one image, highlight the target, have richer details than a single image, and have the ability to withstand harsh environments. Therefore, the fusion of infrared and visible light images aims to carefully fuse the infrared and visible light images in the same scene, while retaining the highlighted targets with thermal radiation information in the infrared images and the high-resolution background texture details in the visible light images, so that The final fused image is more information-rich, which is more conducive to human eye recognition and machine automatic detection, human observation and aesthetics, and computer subsequent image processing.

Existing technology and its shortcomings.

The general steps of image fusion are feature extraction, feature fusion and feature reconstruction. Feature reconstruction is the reverse process of feature extraction. Feature extraction and fusion are the two most critical elements in image fusion. Among traditional methods, multi-scale transform (MST) is the most commonly used image fusion method. Its main feature is that it can accurately characterize the spatial structure of the image and has spatial and spectral consistency. And many multi-scale transformations have been proposed, such as pyramid transform, wavelet transform, contour transform and related variants, etc. In addition, fusion algorithms based on Sparse Representation (SR) and subspace-based methods such as principal component analysis and independent component analysis have also been proposed.

In recent years, deep learning has demonstrated state-of-the-art performance in various fields and has also been successfully applied to image fusion. These algorithms can be roughly divided into three categories, Auto encoder (AE)-based methods, CNN-based methods, and GAN-based methods. Li et al. proposed a simple autoencoder (AE) fusion architecture, which includes an encoder, fusion layer, and decoder. Later, they also increased the complexity of the encoder and proposed a nested fusion method based on autoencoders to obtain more comprehensive feature fusion. The disadvantage of the above method is that it relies on manual design of fusion strategies, which limits the fusion performance. Zhang et al. developed a general image fusion framework through a general network structure, namely feature extraction layer, fusion layer and image reconstruction layer, to learn feature extraction, feature fusion and image reconstruction under the guidance of a class of complex loss functions. This type of method only focuses on global level fusion and does not highlight the target area of interest. Ma et al. creatively introduced GAN into the image fusion community, which utilizes the discriminator to force the generator to synthesize fused images with rich textures. In order to improve the quality of detail information and sharpen the edges of hot targets, they also introduced detail loss and edge enhancement loss. Due to the difficulty of GAN training, this method fails to obtain better fusion quality and cannot highlight salient information.

Contents of the invention

In order to overcome the above shortcomings of the prior art, the purpose of the present invention is to provide an infrared and visible light image fusion method with regional attention to solve the image fusion problem of infrared and visible light images in insufficient illumination scenes. The method proposed by the present invention can give full play to the advantages of infrared and visible light images in scene representation. By extracting and fusing high-dimensional features of images, it is possible to fully integrate infrared thermal radiation information and texture information of visible light images in scenes with insufficient lighting. Moreover, the regional attention module in the fusion network can pay attention to significant areas in high-dimensional features, such as highlighted targets in infrared images and areas with sufficient exposure in visible light images, and increase the pixel intensity of this part during the fusion to achieve regional Attentional image fusion achieves the complementary advantages of infrared and visible light images.

In order to achieve the above objects, the technical solution adopted by the present invention is:

An infrared and visible light image fusion method with regional attention, including:

Step 1, train an autoencoder (Auto Encoder), which includes an encoder and a decoder;

Step 1.1: Read the image I in the training set in RGB format, adjust the image size, and convert it to YCbCr color space;

Step 1.2: Input the brightness channels I _Y of the image to the encoder to obtain the high-dimensional feature map F;

Step 1.3: Input the high-dimensional feature map F to the decoder and output the brightness channel map O _Y ;

Step 1.4: Calculate the feature loss between I _Y and O _Y according to the loss function, then optimize the gradient and backpropagate, and update the model parameters of the autoencoder;

Step 1.5: Repeat steps 1.1 to 1.4 until the number of iterations on the entire training set reaches the set threshold, and the trained autoencoder is obtained;

Step 2: Create a fused image training set

Obtain the infrared and visible light image pairs used for training, and perform sub-image cropping to expand the data set. The cropping size is consistent with the image size adjusted in step 1, and the fused image training set is obtained;

Step 3: Train the fusion network

Step 3.1: Convert the infrared and visible light image pairs (I _R , I _V ) in the fused image training set to YCbCr color space respectively, and extract their respective brightness channel maps to obtain (I _RY , I _VY );

Step 3.2: Input (I _RY , I _VY ) into the encoder trained in step 1 respectively, and calculate the feature map ( _FR , F _V );

Step 3.3: Connect (F _R , F _V ) in the feature dimension, input it into the fusion network, and calculate the fusion feature map F _F ;

Step 3.4: Decode the F _F input decoder to obtain the fusion image O _FY of the brightness channel;

Step 3.5: Calculate the loss value according to the loss function, then optimize the gradient and backpropagate, and update the model parameters of the fusion network;

Step 3.6: Repeat steps 3.1 to 3.5 until the number of calculations on the entire fused image training set reaches the set value, and the trained fusion network is obtained;

Step 4, obtain the fused image

Step 4.1: Follow the method from steps 3.1 to 3.4 for the infrared and visible light images to be fused to obtain the fusion image O _FY of the brightness channel;

Step 4.2: Connect the CbCr channels of O _FY and visible light images in the feature dimension to obtain an image in YCbCr format, and then convert it to RGB format to obtain a fused image.

In one embodiment, the encoder has four convolutional layers and uses dense connections; the decoder uses four convolutional layers for direct connection.

In one embodiment, the convolution kernel size of the encoder and decoder is 3×3, the step is 1, the padding is 1, and the ReLu activation function is used. In step 1.2, the input size is 256×256×1, and the size of the obtained high-dimensional feature map F is 256×256×128. In step 1.3, the size of the brightness channel map O _Y is 256×256×1.

In one embodiment, after step 1.5, change the training data to test data, perform steps 1.1 to 1.3 to obtain O _Y , and then connect O _Y with the CbCr channel in step 1.1 in the feature dimension to obtain an image in YCbCr format, Then convert it to RGB format to obtain the output image O; subjectively verify whether O is consistent with I.

In one embodiment, the calculation steps of step 3.3 are as follows:

(1) Connect (F _R , F _V ) in the feature dimension, and calculate the global information fusion feature map F _{F_0} through the convolution layer Conv_1, convolution layer Conv_2 and convolution layer Conv_3;

(2) Input (F _R , F _V ) into the same regional attention module RAB, and calculate the attention feature map ( _MR , M _V ); connect ( _MR , M _V ) in the feature dimension and input it to Convolutional layer Conv_Att, obtain the fused attention feature map M _RV ;

(3) Calculate the fusion feature map F _F =F _{F_0} +M _RV , that is, add the pixel values at the corresponding positions.

In one embodiment, the Adam optimizer is used to optimize the gradient in both steps 1.4 and 3.5. In step 3.5, the model parameters of the autoencoder are fixed and only the model parameters of the fusion network are updated.

In one embodiment, the step 2 is to select images containing poorly illuminated scenes and significant objects from the public data set TNO to form a training set and a test set, and expand the training set offline by performing the expansion on the original infrared and visible light images. Carry out sub-picture cropping, the sub-picture size is 256×256, and the cropping step is 16.

Compared with the prior art, the beneficial effects of the present invention are:

First: In scenes with insufficient lighting, the texture information of visible light images and the thermal radiation information of infrared images can be fully integrated. After training, the encoder can fully extract the high-dimensional features of the image, and thanks to the high-dimensional feature calculation loss, it ensures that the features of each dimension can be deeply fused during the fusion.

Second, on the basis of fusing the global content, it can also focus on significantly highlighted areas in the source image so that they remain highlighted in the fused image. The fusion network contains two fusion paths, global fusion and salient area fusion. The regional attention module can extract the salient areas of the image from multiple scales, and add the results of the two fusion paths to make the salient areas have higher intensity brightness values to achieve the effect of highlighting.

Third, the fused image has good contrast and clarity. During training, structural loss is measured from three aspects: grayscale, contrast and structural similarity. Gradient loss can make the fused image have good image texture details and increase clarity. In addition, the strategy of fusing only the image brightness channel enables the present invention to process both grayscale and color images. Since the CbCr channel of the visible light image does not participate in the calculation, the fusion result can well restore the color of the visible light image.

Description of the drawings

Figure 1 is the overall framework diagram of the given scheme. The input is the infrared and visible light images to be fused, and the output is the fused image. The network structure consists of the encoder Encoder, the attention fusion network Attention FusionNet and the decoder Decoder. In the dotted box, it is noted that the loss function consists of three parts, namely feature loss, structural similarity loss ssim loss, and gradient loss.

Figure 2 gives the structure of the autoencoder and the composition of the loss function required for training.

Figure 3 gives the structure of the attention fusion network. The input is the feature map (F _R ,F _V ), and the output is the fused feature map F _F .

Figure 4 Network structure of the given regional attention module. Input feature map F and output attention map M.

Figure 5 gives three groups of fused image cases. The boxes marked are the fusion effects of salient targets.

Detailed ways

The embodiments of the present invention will be described in detail below with reference to the drawings and examples.

Visible light sensors can usually capture images that are clear enough and consistent with the observation habits of the human eye when there is sufficient lighting. The areas where the advantages of merging infrared and visible light images are most highlighted are often scenes with insufficient lighting. How to make the fusion result make up for the disadvantage of insufficient exposure and highlight the target of interest, so as to be more conducive to human eye observation and subsequent advanced tasks, is the current problem.

Most previous fusion methods design fusion strategies from a global perspective, focusing on the fusion of image texture details and other content. However, for the salient targets originally in infrared images, such as people, cars, etc., due to the introduction of visible light images in the fusion image components resulting in a decrease in brightness. Although some methods introduce attention to salient targets, they require additional algorithms to obtain binary images of target segmentation in advance. On the other hand, existing methods have insufficient research on nighttime scenes where infrared imaging is most widely used.

Based on this, the present invention provides an infrared and visible light image fusion method with regional attention. The overall architecture is shown in Figure 1, and the steps are as follows:

Step 1, train the autoencoder (Auto Encoder). The structure of the autoencoder is shown in Figure 2, which includes an encoder and a decoder. Each rectangle in the figure represents a layer, and the encoder Encoder and decoder Decoder are composed of convolutional layers and activation layers. The losses include structure loss ssim loss and content loss pixel loss. In this embodiment, the encoder Encoder has four convolution layers and uses dense connections; the decoder Decoder uses four convolution layers to directly connect. The convolution kernel size of the convolution layer is 3×3, step is 1, and padding is 1. The activation layer uses the ReLu activation function. The specific settings of each layer parameter of the encoder Encoder and decoder Decoder are:

Layer Encoder Decoder L1 Conv(I1,O32,K3,S1,P1),ReLu Conv(I128,O64,K3,S1,P1),ReLu L2 Conv(I32,O32,K3,S1,P1),ReLu Conv(I64,O32,K3,S1,P1),ReLu L3 Conv(I64,O32,K3,S1,P1),ReLu Conv(I32,O16,K3,S1,P1),ReLu L4 Conv(I96,O32,K3,S1,P1),ReLu Conv(I16,O1,K3,S1,P1),ReLu

Step 1.1 Use OpenCV's imread function to read the image I in the training set. The read image I is in RGB format, and its size is adjusted to 256×256×3. Then convert from RGB to YCbCr color space. The conversion method can use the OpenCV library function cvtColor. Finally, each pixel of the image is divided by 255 to normalize the pixel value to [0,1] to obtain the input image.

Step 1.2: Input the brightness channel I _Y of the image into the encoder Encoder, the input size is 256×256×1, and the high-dimensional feature map F is obtained, the size is 256×256×128.

Step 1.3: Input the high-dimensional feature map F into the decoder Decoder to obtain the output brightness channel map O _Y with a size of 256×256×1.

Step 1.4: Calculate the feature loss between I _Y and O _Y according to the loss function. The loss function is defined as:

Among them, μ(1-SSIM(O _Y ,I _Y )) is the structural loss, and SSIM(·) is the structural similarity function. is the content loss, that is, calculating the Euclidean distance between I _Y and O _Y. μ is a hyperparameter used to balance the two losses. H and W are the height and width of the image respectively.

Step 1.5: Use Adam optimizer and other methods to optimize the gradient and backpropagate it, and update the model parameters of the autoencoder.

Step 1.6: Repeat steps 1.1 to 1.5. Until the number of iterations on the entire training set reaches the set threshold, the trained autoencoder is obtained.

This embodiment uses the open source color image data set MS-COCO, which contains a total of 80,000 images. Use python and pytorch to implement the algorithm, and train based on a NVIDIA TITAN V GPU. The epoch is set to 2, the batch size is set to 16, and the hyperparameter μ is set to 1.

Step 1.7: In order to verify the above training, the training data can be changed into test data, and steps 1.1 to 1.3 can be performed to obtain O _Y . Then connect O _Y with the CbCr channel map in step 1.1 in the feature dimension to obtain an image in YCbCr format, and then convert it to RGB format to obtain the output image O; subjectively verify whether the output image O is consistent with the input image I.

Step 2: Create a fused image training set and test set.

From the public infrared and visible light image fusion data set TNO, images containing poorly illuminated scenes and salient objects are selected to form a training set and a test set. This embodiment selects 41 pairs of images with darker brightness as the training set and 25 pairs as the test set. Then, the training set is expanded offline. The expansion method is: sub-image cropping of the original infrared and visible light images. The size of each sub-image is consistent with the image size adjusted in step 1, that is, 256×256, and the cropping movement step is 16 , a total of 13940 pairs of infrared and visible light images were finally obtained.

Step 3: Train the fusion network.

Step 3.1: Read the infrared and visible light image pairs (I _R , I _V ) in the fused image training set, and then perform the same operations as in step 1.1 respectively, that is, convert to YCbCr color space, and extract their respective brightness channel maps, Get (I _RY ,I _VY ).

Step 3.2: Input (I _RY , I _VY ) into the encoder Encoder trained in step 1, and calculate the feature map ( _FR , F _V );

Step 3.3: Connect (F _R , F _V ) in the feature dimension, input it into the fusion network, and calculate the fusion feature map F _F . The structure of the fusion layer is shown in Figure 3. The fusion process is divided into two paths, global information fusion and attention feature map fusion, namely global information fusion network and attention feature map fusion network. The former includes three convolutional layers, namely Conv_1, Conv_2 and Conv_3. The latter includes a regional attention module RAB and a convolutional layer Conv_Att. In this embodiment, the network layer parameters can be set as:

The calculation steps in the fusion network are as follows:

(1) Connect (F _R , F _V ) in the feature dimension, and then calculate the global information fusion feature map F _{F_0} through Conv_1, Conv_2 and Conv_3.

(2) Calculate the attention feature map.

Input (F _R , F _V ) into the regional attention module RAB respectively to obtain the attention feature map ( _MR , M _V ) with a size of 256×256×128. Note that the same RFB module is used for separate calculations here. The structure of the regional attention module RAB is shown in Figure 4. These include max pooling, global average pooling, fully connected layers, activation layers, upsampling operations and normalization operations. Represents the weight and feature map multiplied. /> Represents the addition of feature maps. In order to extract feature map weights from multiple scales, the module uses three maximum pooling kernels.

The specific calculation steps are: input the feature map F, perform maximum pooling, and obtain the feature map f _s with a size of Among them, H and W represent the image size. In this example, H and W are both 256, s is 1, 2, and 4 respectively, which means that the sizes of the pooling kernels are 1×1, 2×2, and 4×4 respectively. Then, a global average pooling operation is performed on f _s to obtain a vector with a dimension of 1×1×128. Then connect a fully connected layer and an activation layer, and finally get the weight vector ω _s with a dimension of 1×1×128. The weight value of the k-th dimension feature is used/> Represents, used to measure the kth feature layer/> importance. On the other hand, in order to obtain a feature map of the same size as F, perform an upsampling operation on f _s , and then multiply ω _s and the upsampled feature map in the corresponding dimensions to obtain a weighted feature map/> Among them, k represents the feature map of the kth dimension, and H _up (·) represents the upsampling function. Finally, the feature maps of the three scales are added and then standardized to obtain an attention feature map with a dimension of H×W×128:/> Among them, σ(·) represents the normalization operation.

(3) Connect ( _MR , M _V ) in the feature dimension and input it to the convolution layer Conv_Att to obtain the fused attention feature map M _RV with a size of H×W×128.

(4) Calculate the final fusion feature map F _F =F _{F_0} +M _RV , that is, add the pixel values at the corresponding positions.

Step 3.4: Decode the F _F input decoder to obtain the fusion image O _FY of the brightness channel.

Step 3.5: Calculate the loss value according to the loss function L, use Adam optimizer and other methods to optimize the loss gradient and back propagate, and update the model parameters of the fusion network. Note that the model parameters of the autoencoder are fixed here, and only the model of the fusion network is updated. parameter.

The loss function L contains three parts, the structure loss L _ssim , the feature loss L _pixel and the gradient loss L _gradient . The calculation formula is:

L＝ωL _ssim +λL _pixel +L _gradient

Among them, ω and λ are hyperparameters, used to balance various losses.

The calculation formula of structural loss L _ssim is:

L _ssim =δ(1-SSIM(I _RY ,O _Y ))+(1-δ)(1-SSIM(I _VY ,O _Y ))

Among them, δ is a hyperparameter used to balance the two loss values.

The calculation formula of feature loss is:

Among them, eta is a hyperparameter, and the feature map size is H×W×C. ||·|| ₂ means finding the Euclidean distance of the feature map.

The gradient loss L _gradient calculation formula is:

in, Represents the Sobel gradient calculation operation, used to measure the fine-grained texture information of the image.

Step 3.6: Repeat steps 3.1 to 3.5 until the number of iterations reaches the set threshold on the entire fused image training set, thereby obtaining the trained fusion network. In this embodiment, the training is based on an NVIDIA TITAN V GPU, using the Adam optimizer, and the batch size and epoch are 4 and 2 respectively. The initial learning rate is set to 1×10 ^-4 , and the hyperparameters ω, λ, δ, and eta of the loss function are set to 1, 2.7, 0.5, and 0.5 respectively.

Step 4: Enter the test data and obtain the fused image.

Step 4.1: Use the test data or the infrared and visible light images to be fused according to the method from steps 3.1 to 3.4 to obtain the fusion image _OFY of the brightness channel.

Step 4.2: Connect O _FY and the CbCr channel of the visible light image in the feature dimension to obtain an image in YCbCr format, and then convert it to RGB format to obtain a fused image.

Three sets of fused images were selected from the test, as shown in Figure 5. It can be seen from the figure that the fused image incorporates the texture details of the visible light image, as shown in the dotted box in the figure, and the overall brightness of the image is improved to a certain extent. At the same time, the salient areas only in the infrared image are also well reflected in the fused image, as shown by the solid line box in the figure.

Claims (6)

1. A method for fusing infrared and visible light images with regional attention, comprising:

step 1, training a self-encoder, wherein the self-encoder comprises an encoder and a decoder;

step 1.1: reading an image I in a training set in an RGB format, adjusting the size of the image, and converting the image into a YCbCr color space;

step 1.2: luminance channel I of image _Y Inputting the high-dimensional characteristic diagram F into an encoder to obtain a high-dimensional characteristic diagram F;

step 1.3: inputting the high-dimensional characteristic diagram F to a decoder to output a brightness channel diagram O _Y ；

Step 1.4: calculation of I from loss function _Y And O _Y The characteristic loss between the two is then optimized, gradient is propagated in the opposite direction, and model parameters of the self-encoder are updated;

step 1.5: repeating the steps 1.1 to 1.4 until the iteration times on the whole training set reach a set threshold value, and obtaining a trained self-encoder;

step 2: creating a fused image training set

Acquiring an infrared and visible light image pair for training, and carrying out subgraph cutting expansion data set, wherein the cutting size is consistent with the image size adjusted in the step 1, so as to obtain a fused image training set;

step 3: training fusion network

Step 3.1: the infrared and visible light image pair (I _R ,I _V ) Respectively converting into YCbCr color space, and respectively extracting respective brightness channel diagrams to obtain (I) _RY ,I _VY )；

Step 3.2: respectively (I) _RY ,I _VY ) Inputting the trained encoder in step 1, and calculating to obtain a feature map (F _R ，F _V )；

Step 3.3: will (F) _R ，F _V ) Connecting in the feature dimension, inputting into a fusion network, and calculating to obtain a fusion feature map F _F The method comprises the following steps:

(1) Will (F) _R ，F _V ) In the feature dimension connection, the global information fusion feature diagram F is obtained through calculation of a convolution layer Conv_1, a convolution layer Conv_2 and a convolution layer Conv_3 _{F_0} ；

(2) Respectively (F) _R ，F _V ) Inputs the same regional attention module RAB, calculates to obtain attention characteristic diagram (M _R ,M _V ) The method comprises the steps of carrying out a first treatment on the surface of the Will (M) _R ,M _V ) Connected in the feature dimension, input to the convolution layer Conv_Att to obtain a fused attention feature map M _RV The method comprises the steps of carrying out a first treatment on the surface of the The regional attention module RAB comprises a maximum pooling, a global average pooling, a full connection layer, an activation layer, an up-sampling operation and a standardization operation;

(3) Calculate the fusion feature map F _F ＝F _{F_0} +M _RV I.e. the corresponding position pixel values are added;

step 3.4: will F _F Decoding by an input decoder to obtain a fusion image O of the brightness channel _FY ；

Step 3.5: calculating a loss value according to the loss function, optimizing the gradient, and reversely propagating to update the model parameters of the fusion network;

step 3.6: repeating the steps 3.1 to 3.5 until the calculation times on the whole fusion image training set reach a set value, and obtaining a trained fusion network;

step 4, obtaining a fusion image

Step 4.1: the infrared and visible light image pair to be fused is subjected to the method of steps 3.1 to 3.4 to obtain a fused image O of the brightness channel _FY ；

Step 4.2: o is added with _FY And connecting the CbCr channels of the visible light image in characteristic dimension to obtain an image in YCbCr format, and converting the image into RGB format to obtain a fusion image.

2. The method of claim 1, wherein the encoder has four convolutions and uses dense connections; the decoder is directly connected by adopting four convolution layers; in the encoder and decoder, the convolution kernel size is 3×3, step is 1, padding is 1, and the ReLu activation function is used.

3. The method for fusing infrared and visible light images having regional attention as defined in claim 2, wherein said step 1.2 is a step of inputting 256×256×1, obtaining a high-dimensional feature map F of 256×256×128, and said step 1.3 is a step of luminance channel map O _Y The size is 256×256×1.

4. The method of image fusion of infrared and visible light with regional attention according to claim 1, wherein after step 1.5, the training data is changed to test data, steps 1.1 to 1.3 are performed to obtain O _Y Then O is taken _Y Connecting the images with the CbCr channels in the step 1.1 in characteristic dimension to obtain images in a YCbCr format, and converting the images into an RGB format to obtain an output image O; subjective verification O was consistent with I.

5. The method for fusing infrared and visible light images with regional attention according to claim 1, wherein the step 1.4 and the step 3.5 are both to optimize the gradient by adopting an Adam optimizer, wherein the step 3.5 is to fix the model parameters of the self-encoder and update only the model parameters of the fused network.

6. The method according to claim 1, wherein the step 2 is to select images with significant objects and containing insufficient illumination scenes from the public dataset TNO to form a training set and a test set, and to expand the training set offline in such a way that the original infrared and visible images are sub-clipped, the sub-clip size is 256×256, and the clipping step size is 16.