CN115965557A - Polarization recovery imaging method based on transformation Mueller matrix network - Google Patents

Abstract

The invention discloses a polarization recovery imaging method based on a transformation Mueller matrix network, which comprises the following steps: shooting a clear polarization image of an underwater target and polarization images of the target under different turbidity degrees; and secondly, aiming at the loss of spatial resolution brought by using a polarization camera with a split focal plane, converting a clear polarization image of an underwater target into a clear intensity image of the underwater target by using a targeted interpolation method, constructing a transformation Mueller matrix network by combining an inverted residual convolution structure and a channel attention mechanism according to the idea of the Mueller matrix, establishing a data set in a simulated environment, training the network by fully utilizing polarization information of the image to construct a network capable of outputting a corresponding transformation Mueller matrix according to the input foggy or turbid polarization image, repairing the image by using the transformation Mueller matrix output by the network to obtain a good recovery effect, and keeping stable performance under various environments.

Description

A Polarization Restoration Imaging Method Based on Transformed Mueller Matrix Network

technical field

The invention relates to the technical field of polarization imaging, in particular to a polarization restoration imaging method based on a transformed Muller matrix network.

Background technique

Nowadays, computer vision technology has brought great help to all walks of life. Whether it is traffic monitoring on the road, image cameras for self-driving vehicles, or cameras for underwater exploration equipment, it is necessary to acquire images and pass Computer vision technology analyzes and makes judgments about the situation. However, whether it is in the air or in the underwater medium, impurities will seriously interfere with the images imaged by the camera, such as low contrast, low brightness, and blurred object details. This makes image-based object detection and analysis difficult to use underwater. At present, the restoration methods of foggy and turbid images are mainly divided into two categories, including restoration methods based on physical models and image enhancement methods based on non-physical models. The restoration method based on the physical model needs to consider the degradation process of the image, mathematically model the aerosol or underwater imaging process, estimate the model parameters to restore the image through inverse deduction, or use deep learning technology to learn the mapping function and then restore the image. Image enhancement methods based on non-physical models mainly include histogram equalization methods, color correction methods, and fusion-based methods. Both technologies improve the image to a certain extent and improve the image quality. Relatively speaking, image enhancement generally enhances the visual experience, which is biased towards human subjective judgment, and the lost details will not be repaired. The image restoration method is to carry out image modeling according to the image degradation model, and design a mapping function to restore the original image. Polarization imaging technology is an image restoration technology that has been widely studied at present, but its imaging effect still has certain limitations. When it is applied in different environments, the imaging effect is uneven, the image quality is not clear enough, and the applicability is not wide enough, which affects further research. Applications.

Contents of the invention

In order to overcome the defects in the above-mentioned prior art, the present invention provides a polarization restoration imaging method based on the transformed Muller matrix network, which establishes a data set in a simulated environment, makes full use of the polarization information of the image, and trains the network to construct A network that can output the corresponding transformed Mueller matrix according to the input foggy or turbid polarized image. Using the transformed Mueller matrix output by the network to repair the image can achieve a good restoration effect, and it can be maintained in a variety of environments. stable performance.

Technical solutions

A polarization restoration imaging method based on transformed Muller matrix network, comprising the following steps:

Step 1. Take clear polarization images of underwater targets and polarization images of targets at different concentrations;

Step 2. In view of the spatial resolution loss caused by the use of the sub-focal plane polarization camera, a targeted interpolation method is used to convert the clear polarization image of the underwater target object into a clear intensity image of the underwater target object;

Step 3. According to the idea of transforming the Muller matrix, construct the inverted residual convolutional neural network. The overall network structure is designed according to the U-shaped structure. After the feature map is input, it will first pass through four encoders. A differential polarization attention channel module and a downsampling layer to extract features, followed by upsampling through a quadruple decoder module that also contains an inverted residual polarization attention channel module and an upsampling layer that contains channel attention The force module is used to extract the polarization image from different channels and give different weights to different channels; the inverted residual structure is used to reduce the loss of high-dimensional information after passing through the activation function, and the DW convolution and PW convolution modules are used It is used to reduce the computational load of the network and optimize network performance. The loss function of the network includes three parts: edge loss, content loss, and pixel loss. The total loss function is obtained by weighting edge loss, content loss, and pixel loss.

Step 4. Train the network; use the data processed in step 2 to train the inverted residual convolutional neural network constructed in step 3. The trained inverted residual convolutional neural network can The repaired Mueller matrix corresponding to the polarized image output;

Step 5: Restoring the image; using the repaired Muller matrix outputted by the inverted residual convolutional neural network trained in Step 4 to repair the foggy or turbid polarized image, and finally obtain a significantly improved restored image.

Further, in step 1, the clear polarization image of the underwater target is taken: the light beam emitted by the light source passes through the polarizer of the polarization modulation system and the beam expander in sequence, and then irradiates the target in the water, and reaches the branch after being reflected by the target. The focal plane polarization camera can obtain a clear polarization image of the target object.

Further, the polarized images of the target object under different turbidity in step 1 are taken: gradually add skim milk to the water, and use a 5-megapixel polarized Gigabit Ethernet industrial camera to shoot the target object under different turbidity concentration environments, thereby The polarization images of the target object in the turbid underwater environment under different polarization angles and different turbidity concentrations are obtained.

Further, a 532nm blue-green laser is used as the light source.

Further, the inverted residual polarized attention channel module in step 3 consists of two parts, the first part is an inverted residual structure composed of three convolutional layers, which contains a 1*1 convolution Upscale the feature map, a 3*3 convolution extracts the feature, a 1*1 convolution reduces the dimensionality of the feature map, and finally includes a BN layer for normalization. The second part is a The channel attention structure, which includes an average pooling layer, a one-dimensional convolutional layer, and a Sigemoid activation function, avoids dimension reduction and effectively performs cross-channel interaction.

Furthermore, the inverted residual convolutional neural network constructed in step 3 is based on the idea of transforming the Muller matrix. The inverted residual polarization attention channel module of the network can extract the corresponding feature information so that the network can be based on the corresponding The feature information finds the relationship between the Mueller matrix of the turbid image and the Mueller matrix of the clear polarized image, and outputs the transformed Mueller matrix corresponding to the turbid image accordingly.

Further, the restoration of the Mueller matrix in step 5 is a physical model proposed based on the idea of the Mueller matrix, which describes the transformation relationship of the Mueller matrix of the clear image and the turbid image.

Further, the target is placed in a glass cylinder filled with water to simulate an underwater environment.

Further, the glass cylinder is made of PMMA (polymethyl methacrylate).

Beneficial effect

Compared with the prior art, the present invention has the following beneficial effects:

According to the idea of Mueller matrix, combined with the inverted residual convolution structure and channel attention mechanism, a transformed Mueller matrix network is constructed, a data set is established in a simulated environment, and the polarization information of the image is fully utilized to train the network to build a network that can According to the input foggy or turbid polarized image, the corresponding inpainting Muller matrix is output. Using the inpainting Mueller matrix to inpaint the image can achieve a good restoration effect, and can maintain stable performance in various environments.

Description of drawings

Fig. 1 is a schematic diagram of the overall process of a polarization restoration imaging method based on a transformed Muller matrix network in the present invention;

Fig. 2 is a schematic diagram of the experimental device of the present invention;

Fig. 3 is a schematic diagram of generating network structure;

Figure 4 is a schematic diagram of the structure of the inverted residual polarization channel attention block;

Fig. 5 is the experimental rendering of the transformation Muller matrix network, (a) is a high turbidity image, (b) is an image restored by the present invention, and (c) is a clear intensity image;

Fig. 6 is a table of restoration effect evaluation indexes EME, SSIM, and PSNR.

reference sign

Light source 1, polarizer 2, beam expander 3, glass cylinder 4, underwater environment 5, target object 6, focal plane polarization camera 7.

Detailed ways

In order to better illustrate and set forth the content of the present invention, the following will be expanded and described in conjunction with the accompanying drawings and implementation examples:

As shown in Fig. 1-Fig. 6, the present invention discloses a polarization restoration imaging method based on transformed Muller matrix network, comprising the following steps:

Step 1. Take a clear polarization image of the underwater target object and polarization images of the target object under different turbidity;

Step 2. In view of the spatial resolution loss caused by the use of the sub-focal plane polarization camera, a targeted interpolation method is used to convert the clear polarization image of the underwater target object into a clear intensity image of the underwater target object;

Step 3. According to the idea of transforming the Muller matrix, construct the inverted residual convolutional neural network. The overall network structure is designed according to the U-shaped structure. After the feature map is input, it will first pass through four encoders. A differential polarization attention channel module and a downsampling layer to extract features, followed by upsampling through a quadruple decoder module that also contains an inverted residual polarization attention channel module and an upsampling layer that contains channel attention The force module is used to extract the polarization image from different channels and give different weights to different channels; the inverted residual structure is used to reduce the loss of high-dimensional information after passing through the activation function, and the DW convolution and PW convolution modules are used It is used to reduce the computational load of the network and optimize network performance. The loss function of the network includes three parts: edge loss, content loss, and pixel loss. The total loss function is obtained by weighting edge loss, content loss, and pixel loss.

Step 4. Train the network; use the data processed in step 2 to train the inverted residual convolutional neural network constructed in step 3. The trained inverted residual convolutional neural network can The repaired Mueller matrix corresponding to the polarized image output;

Step 5: Restoring the image; using the repaired Muller matrix outputted by the inverted residual convolutional neural network trained in Step 4 to repair the foggy or turbid polarized image, and finally obtain a significantly improved restored image.

Further, in step 1, the clear polarized image of the underwater target is taken: the light beam emitted by the light source 1 sequentially passes through the polarizer 2 of the polarization modulation system and the beam expander 3, and then irradiates the target 6 in the water, passes through the target 6 reaches the focal plane polarization camera 7 after reflection, thereby obtaining a clear polarization image of the target object 6 .

Further, in step 1, take the polarization images of the target objects at different concentrations: gradually add skim milk to the water, and use a 5-megapixel polarized Gigabit Ethernet industrial camera to shoot the target objects 6 in different turbidity concentration environments, thereby The polarization images of the target object 6 in the turbid underwater environment at different polarization angles and different turbidity concentrations are obtained.

Further, a 532nm blue-green laser is used as the light source 1 .

Further, the inverted residual polarized attention channel module in step 3 consists of two parts, the first part is an inverted residual structure composed of three convolutional layers, which contains a 1*1 convolution Upscale the feature map, a 3*3 convolution extracts the feature, a 1*1 convolution reduces the dimensionality of the feature map, and finally includes a BN layer for normalization. The second part is a The channel attention structure, which includes an average pooling layer, a one-dimensional convolutional layer, and a Sigemoid activation function, avoids dimension reduction and effectively performs cross-channel interaction.

Furthermore, the inverted residual convolutional neural network constructed in step 3 is based on the idea of transforming the Muller matrix. The inverted residual polarization attention channel module of the network can extract the corresponding feature information so that the network can be based on the corresponding The feature information finds the relationship between the Mueller matrix of the turbid image and the Mueller matrix of the clear polarized image, and outputs the transformed Mueller matrix corresponding to the turbid image accordingly.

Further, the restoration of the Mueller matrix in step 5 is a physical model proposed based on the idea of the Mueller matrix, which describes the transformation relationship of the Mueller matrix of the clear image and the turbid image.

Further, the target object 6 is placed in a glass cylinder 4 filled with water for simulating an underwater environment 5 .

Further, the glass cylinder 4 is made of PMMA (polymethyl methacrylate).

Specifically, step 1, using an underwater active imaging system, adopts linearly polarized light for active illumination, and takes clear polarization images of the underwater target 6 and polarization images of the target 6 at different concentrations. In this embodiment, 532nm blue Green laser is used as light source 1, and PMMA (polymethyl methacrylate) glass cylinder 4 is selected;

The light beam emitted by the light source 1 sequentially passes through the polarizer 2 of the polarization modulation system and the beam expander 3, and then irradiates the target object 6 in the water. After being reflected by the target object 6, it reaches the sub-focus plane polarization camera 7, and takes pictures of the underwater target object. 6 clear intensity images, gradually add skim milk to the water, and use a 5-megapixel polarized Gigabit Ethernet industrial camera to shoot the target object 6 in different turbidity concentration environments; thus the target object 6 in the turbid underwater environment is obtained. Polarization angles and polarization images under different turbidity concentrations. In this embodiment, 20 groups of images are taken, and 20 different concentrations of milk are added to each group to simulate the underwater environment 5 under 20 different concentrations. Each concentration is based on The shooting method in step 1 takes a polarized image of the target object 6 (including "mosaic" images with polarization directions of 0°, 45°, 90° and 135°, and every 4 pixels in this image contain images from 4 pixels in the polarization direction);

Step 2, establish a data set; crop the image obtained in step 1, expand the data set by flipping and rotating, and divide it into a training set, a verification set, and a test set in a ratio of 8:1:1;

Step 3. Construct the inverted residual polarization channel attention module. The inverted residual polarization attention channel module consists of two parts. The first part is an inverted residual structure composed of three convolutional layers, which contains an 11 Convolution increases the dimension of the feature map, a 33 convolution extracts the feature, a 1*1 convolution reduces the dimension of the feature map, and finally includes a BN layer for normalization. The second part is a Channel attention structure, which includes an average pooling layer, a one-dimensional convolutional layer, and a Sigemoid activation function, which avoids dimension reduction and effectively performs cross-channel interaction;

Step 4. According to the idea of transforming the Mueller matrix, construct the inverted residual convolutional neural network. The overall network structure is designed according to the U-shaped structure. After the feature map is input, it will first pass through four encoders, and the encoder contains the inverted residual Differential polarization attention channel module and downsampling layer to extract features, and then upsampled by quadruple decoder module, which also includes inverted residual polarization attention channel module and upsampling layer, which contains channel attention module It is used to extract polarization images from different channels and give different weights to different channels. The inverted residual structure is used to reduce the loss of high-dimensional information after passing through the activation function. The DW convolution and PW convolution modules are used for To reduce the computational load of the network and optimize network performance, the loss function of the network includes three parts: edge loss, content loss, and pixel loss. The total loss function is obtained by weighting the above three losses.

It can be seen from the experimental results that the present invention can effectively restore the polarization images taken in high-concentration muddy water, combined with the objective evaluation standard EME (the value of measure of enhancement), structural similarity SSIM (Structure Similarity Index Measure), PSNR (Peak Signal to Noise Ratio) and subjective experience, the image restoration effect is remarkable.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the technical solutions of the present invention have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that it still The technical solutions described in the foregoing embodiments can be modified, or some of the technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (9)

1. A polarization restoration imaging method based on transformation Muller matrix network, is characterized in that, comprises the following steps: Step 1. Take a clear polarization image of the underwater target object and polarization images of the target object under different turbidity; Step 2. In view of the spatial resolution loss caused by the use of the sub-focal plane polarization camera, a targeted interpolation method is used to convert the clear polarization image of the underwater target object into a clear intensity image of the underwater target object; Step 3. According to the idea of transforming the Muller matrix, construct the inverted residual convolutional neural network. The overall network structure is designed according to the U-shaped structure. After the feature map is input, it will first pass through four encoders. A differential polarization attention channel module and a downsampling layer to extract features, followed by upsampling through a quadruple decoder module that also contains an inverted residual polarization attention channel module and an upsampling layer that contains channel attention The force module is used to extract the polarization image from different channels and give different weights to different channels; the inverted residual structure is used to reduce the loss of high-dimensional information after passing through the activation function, and the DW convolution and PW convolution modules are used It is used to reduce the computational load of the network and optimize network performance. The loss function of the network includes three parts: edge loss, content loss, and pixel loss. The total loss function is obtained by weighting edge loss, content loss, and pixel loss. Step 4. Train the network; use the data processed in step 2 to train the inverted residual convolutional neural network constructed in step 3. The trained inverted residual convolutional neural network can The repaired Mueller matrix corresponding to the polarized image output; Step 5: Restoring the image; using the repaired Muller matrix outputted by the inverted residual convolutional neural network trained in Step 4 to repair the foggy or turbid polarized image, and finally obtain a significantly improved restored image. 2. A kind of polarization restoration imaging method based on transformation Muller matrix network according to claim 1, it is characterized in that, the clear polarized image of photographing underwater target object in the step 1: the light beam that sends by light source (1) sequentially After passing through the polarizer (2) of the polarization modulation system and the beam expander (3), the target object (6) in the water is irradiated, and after being reflected by the target object (6), it reaches the sub-focus plane polarization camera (7), thereby obtaining the target The clear polarized image of object (6). 3. a kind of polarization restoration imaging method based on transformation Muller matrix network according to claim 2, is characterized in that, the polarization image of the target object under photographing different turbidity in the step 1: add skimmed milk step by step in water, Using a 5-megapixel polarization Gigabit Ethernet industrial camera to shoot the target object (6) in different turbidity concentration environments, so as to obtain the polarization of the target object (6) in the turbid underwater environment at different polarization angles and different turbidity concentrations image. 4. A polarization restoration imaging method based on a transformed Mueller matrix network according to claim 2 or 3, characterized in that a 532nm blue-green laser is used as the light source (1). 5. a kind of polarization restoration imaging method based on transformation Muller matrix network according to claim 4, is characterized in that, described inverse residual polarization attention channel module in step 3 is made up of two parts, the first Part of it is an inverted residual structure composed of three convolutional layers, which includes a 1*1 convolution to upscale the feature map, a 3*3 convolution to extract features, and a 1*1 convolution The feature map is dimensionally reduced, and finally a BN layer is included for normalization. The second part is a channel attention structure, which includes an average pooling layer, a one-dimensional convolution layer, and a Sigemoid activation function. Dimensionality reduction is avoided, and cross-channel interaction is effectively carried out. 6. a kind of polarization restoration imaging method based on transformation Muller matrix network according to claim 5, it is characterized in that, the inverted residual convolutional neural network constructed in step 3 is based on the thought design of transformation Muller matrix, The inverted residual polarization attention channel module of the network can extract the corresponding feature information so that the network can find the relationship between the Mueller matrix of the turbid image and the Mueller matrix of the clear polarized image according to the corresponding feature information, and output accordingly The transformed Mueller matrix corresponding to the turbid image. 7. A kind of polarization restoration imaging method based on transformation Mueller matrix network according to claim 6, it is characterized in that, described repair Mueller matrix in the step 5 is the physical model that proposes based on the thought of Mueller matrix, describes is the transformation relationship of the Mueller matrix of the clear image and the turbid image. 8. A kind of polarization restoration imaging method based on transformation Muller matrix network according to claim 2 or 3, is characterized in that, described object (6) is placed in the glass tank (4) that fills water, uses for simulating underwater environments (5). 9. A kind of polarization restoration imaging method based on transformation Muller matrix network according to claim 8, is characterized in that, described glass cylinder (4) adopts PMMA (polymethyl methacrylate) material.

Images