CN114529573A - Reflection separation method and system of light field image - Google Patents

Abstract

The invention provides a reflection separation method and system for light field images, which are used to solve the problem of inaccurate reflection separation of light field images in the prior art. The method obtains multi-scale spatial angle features based on the light field image; after the refocusing operation is performed to obtain the refocusing image, a preliminary background image and a preliminary reflection image are respectively generated, and further convolved to obtain a background edge map, a background refocusing image, Reflection edge map, reflection refocusing image; then generate background and reflection spatial attention weights based on spatial attention mechanism map; combine multi-scale spatial angle features and refocusing features to obtain background reconstruction features and reflection reconstruction features, and further generate reflections Channel attention weights, adjust the reconstructed features according to the two weights, and finally use the U-shaped network to reconstruct the separated background image and reflection image respectively. The invention automatically recovers the corresponding background image and the reflection image through the light field image with reflection, and improves the qualitative and quantitative analysis effect of the image.

Description

Method and system for reflection separation of light field images

technical field

The invention belongs to the field of image processing, and in particular relates to a reflection separation method and system for light field images.

Background technique

A light field camera with the addition of a microlens array between the main lens and the imaging sensor. The microlens array is a two-dimensional array composed of many microlens units, which is in a conjugate relationship with the main lens and the imaging sensor, so the position and direction of a beam of light can be determined by the microlens plane and the imaging sensor plane, so that the light Field cameras are capable of simultaneously recording the intensity, position and direction of light. Compared with traditional cameras, it has the characteristics of taking pictures first and then focusing, and has huge advantages in multi-view information collection.

Reflection is a common image disturbance. When a photo is taken through a transparent plane such as glass, objects on the camera side will also be ingested, which is not only visually unsightly, but also affects the performance of some other computer vision tasks. , for example: object detection and object tracking. The light field image captured by the light field camera also has the same problem. Therefore, it is generally necessary to separate the image by a separation method to obtain the actual scene image.

In the prior art, the reflection separation method of a single image usually needs to assume that the reflection layer is relatively weak or that the reflection layer is relatively blurred. These assumptions severely limit the use of these methods in actual complex scenes; at the same time, the reflection separation method based on assumptions The calculation is often performed in minutes, and the processing speed is slow.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to propose a reflection separation method for light field images, obtaining multi-scale space angle features through multi-scale space angle convolution, obtaining preliminary background images and reflection images through refocusing images obtained by refocusing operations, Further, generate the background edge map, background refocusing feature, reflection edge map, and reflection refocusing feature, and then generate the background spatial attention weight and background channel attention weight through the background edge map and background refocusing feature, and adjust the background reconstruction feature. ; Generate reflective spatial attention weights and reflective channel attention weights through reflective edge maps and reflective refocusing features, adjust reflective reconstruction features, and reconstruct background images and reflections respectively according to the adjusted background reconstruction features and reflective reconstruction features Image, through the light field image with reflection, the corresponding background image and reflection image are automatically recovered, which improves the qualitative and quantitative analysis effect of the image.

In order to achieve the above object, the technical scheme adopted in the present invention is as follows:

In a first aspect, an embodiment of the present invention provides a reflection separation method for a light field image, where the reflection separation method includes the following steps:

Step S1, obtain the original light field image, perform multi-scale spatial angle convolution, and obtain multi-scale spatial angle features;

Step S2, obtaining a refocusing image by performing a refocusing operation on the original light field image;

Step S3, performing a dynamic convolution operation on the refocusing image to generate a preliminary background image and a preliminary reflection image respectively;

Step S4, respectively convolving the preliminary background image and the preliminary reflection image to obtain a background edge map, a background refocusing feature, a reflection edge map, and a reflection refocusing feature;

Step S5, based on the spatial attention mechanism, using the background edge map to generate background spatial attention weights, and using the reflected edge map to generate reflective spatial attention weights;

Step S6, merging the multi-scale space angle feature and the background refocusing feature to obtain the background reconstruction feature, and merging the multi-scale space angle feature and the reflection refocusing feature to obtain the reflection reconstruction feature;

Step S7, based on the channel attention mechanism, utilize the background reconstruction feature to generate the background channel attention weight, and utilize the reflection reconstruction feature to generate the reflection channel attention weight;

Step S8, utilize the background space attention weight and the background channel attention weight to adjust the background reconstruction feature, and obtain the adjusted background reconstruction feature; use the reflection space attention weight and the reflection channel attention weight to adjust the reflection Reconstruction features to obtain adjusted reflection reconstruction features;

In step S9, the background image is reconstructed from the adjusted background reconstruction feature by using a U-shaped network, and the reflection image is reconstructed from the adjusted reflection reconstruction feature by using the U-shaped network.

As a preferred embodiment of the present invention, the step S1 multi-scale spatial angle convolution specifically includes the following steps:

Use spatial 2D convolution to perform downsampling operations, using 1 and 2 times respectively, to obtain images of 1/2 original resolution and 1/4 original resolution;

Perform 4 spatial 2D convolutions and 4 angular 2D convolutions on the original, 1/2 original resolution and 1/4 original resolution images respectively to obtain features of different scales;

The features of different scales are fused by spatial 2D convolution to obtain multi-scale spatial angle features.

As a preferred embodiment of the present invention, the formula for the refocusing operation in step S2 is:

In formula (1), d represents the value of the refocusing operation, L represents the light field image, u, v represent the angular dimension, x, y represent the spatial dimension, u ^c , v ^c represent the angular coordinates of the central viewing angle, u∈[0 ,U], v∈[0,V], U and V are two constants, and I _d (x, y) represents the refocusing image after the refocusing operation using numerical values.

As a preferred embodiment of the present invention, the step S3 dynamic convolution specifically includes the following steps:

Input the refocused image into 2 different 3D convolutions to generate background dynamic convolution kernel and reflection dynamic convolution kernel respectively;

generating a preliminary background image from the refocused image using a background dynamic convolution kernel;

A preliminary reflection image is generated from the refocused image using a reflection dynamic convolution kernel.

As a preferred embodiment of the present invention, the step S3 utilizes dynamic convolution to generate the preliminary background image and the preliminary reflection image, which can be expressed by the following formula:

B _inter (x,y)＝∑ _d I(d,x,y)*W _B (d,x,y) (2)

R _inter (x,y)=∑ _d I(d,x,y)*W _R (d,x,y) (3)

In equations (2) and (3), d represents the value of the refocusing operation, x, y represent the spatial dimension, I(d, x, y) represents all refocusing images, and B _inter (x, y) represents the preliminary background Image, WB (d,x,y) represents the background dynamic convolution kernel, R _inter (x,y) represents the preliminary reflection image, and W _{R (} d,x,y) represents the reflection dynamic convolution kernel.

As a preferred embodiment of the present invention, the step S5 is based on the spatial attention mechanism, using the background edge map to generate the background spatial attention weight, and using the reflective edge map to generate the reflective spatial attention weight, which specifically includes the following steps:

Input the background edge map into the spatial 2D convolution, change its channel to 1, and obtain the background spatial attention weight;

The reflection edge map is input into the spatial 2D convolution, and its channel is changed to 1 to obtain the reflection spatial attention weight.

As a preferred embodiment of the present invention, the step S7 is based on the channel attention mechanism, using the background reconstruction feature to generate the background channel attention weight, and using the reflection reconstruction feature to generate the reflection channel attention weight, specifically Contains the following steps:

Perform global maximum pooling and global average pooling operations on the background reconstruction feature, change its spatial resolution to 1*1, and obtain background maximum pooling features and background average pooling features;

Input the background maximum pooling feature and background average pooling feature into the fully connected layer to obtain the background channel attention weight;

Perform global maximum pooling and global average pooling operations on the reflection reconstruction feature, change its spatial resolution to 1*1, and obtain reflection maximum pooling features and reflection average pooling features;

Input the reflection max pooling feature and reflection average pooling feature into the fully connected layer to obtain the reflection channel attention weight.

As a preferred embodiment of the present invention, the step S8 obtains the adjusted background reconstruction features and reflection reconstruction features, and obtains them by the following formula:

表示调整后的背景重构特征；CA_R(x,y)表示反射通道注意力权重，SA_R(x,y)表示反射空间注意力权重，F_R(x,y)表示反射重构特征，

表示调整后的反射重构特征。In equations (4) and (5), x, y represent the spatial dimension, CA _B (x, y) represents the background channel attention weight, SA _B (x, y) represents the background spatial attention weight, F _B (x, y) represents the background reconstruction feature,

Represents the adjusted background reconstruction features; _CAR (x,y) represents the reflection channel attention weight, SAR (x,y) represents the reflection spatial attention weight, F _{R (} x,y) represents the reflection reconstruction feature,

Represents the adjusted reflection reconstruction feature.

As a preferred embodiment of the present invention, the structure of the U-shaped network in the step S9 specifically includes an encoding layer, a decoding layer and a layer-hopping structure; wherein,

The coding layer is composed of 3 groups of modules including spatial 2D convolution;

The decoding layer is composed of 3 groups of modules including spatial 2D convolution and transposed convolution;

The skip layer structure is used to connect the coding layer and the decoding layer.

In a second aspect, an embodiment of the present invention also provides a light field image reflection separation system, the system includes: a light field image acquisition module, a multi-scale spatial angle feature generation module, a refocusing module, a preliminary separation image generation module, Edge map generation module, spatial attention weight generation module, refocusing feature generation module, reconstruction feature generation module, channel attention weight generation module, reconstruction feature adjustment module, and image separation module; among them,

The light field image acquisition module is used to acquire the original light field image;

The multi-scale spatial angle feature generation module is used to perform multi-scale spatial angle convolution on the original light field image, obtain multi-scale spatial angle features, and send them to the reconstructed feature generation module;

The refocusing module is used for performing a refocusing operation on the original light field image to obtain the refocusing image;

The preliminary separated image generation module is used to perform a dynamic convolution operation on the refocusing image to generate a preliminary background image and a preliminary reflection image respectively;

The edge map generation module is used to convolve the preliminary background image and the preliminary reflected image respectively to obtain the background edge map and the reflected edge map;

The spatial attention weight generation module is configured to generate background spatial attention weights by using the background edge map based on the spatial attention mechanism, and generate reflective spatial attention weights by using the reflected edge map;

The refocusing feature generation module is configured to convolve the preliminary background image and the preliminary reflection image respectively to obtain the background refocusing feature and the reflection refocusing feature;

The reconstruction feature generation module is used for combining the multi-scale spatial angle feature and the background refocusing feature to obtain the background reconstruction feature, and combining the multi-scale spatial angle feature and the reflection refocusing feature to obtain the reflection reconstruction feature;

The channel attention weight generation module is used to generate the background channel attention weight by using the background reconstruction feature, and use the reflection reconstruction feature to generate the reflection channel attention weight;

The reconstruction feature adjustment module is used to adjust the background reconstruction feature by using the background space attention weight and the background channel attention weight to obtain the adjusted background reconstruction feature; using the reflection space attention weight and reflection The channel attention weight adjusts the reflection reconstruction feature, and obtains the adjusted reflection reconstruction feature;

The image separation module is used for reconstructing the background image from the adjusted background reconstruction features by using the U-shaped network, and reconstructing the reflection image from the adjusted reflection reconstruction features by using the U-shaped network.

The present invention has the following beneficial effects:

The reflection separation method of the light field image in the embodiment of the present invention uses the light field image with reflection to automatically separate the background image and the reflection image, so as to obtain a clearer and independent background image and reflection image in a real scene, and the image has a richer restoration. details, and the processing speed has also been greatly improved. The embodiments of the present invention can be applied to image preprocessing for various common computer vision tasks, such as: target detection, target tracking, 3D reconstruction, etc., without increasing the image processing time, the image processing effect is improved, and the image quality is improved. Qualitative and quantitative analysis effects, for example, the separated background image can be used for more accurate target detection, target tracking, 3D reconstruction, and the separated reflection image can be used for the detection of cases in the criminal investigation field.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a flowchart of a reflection separation method of a light field image according to an embodiment of the present invention;

FIG. 2 shows an example diagram of the generation of a refocusing image in an embodiment of the present invention;

FIG. 3 is a schematic diagram of reflection separation of a light field image according to an embodiment of the present invention.

Detailed ways

The technical problems, technical solutions and advantages of the present invention will be explained in detail below by referring to the exemplary embodiments. The exemplary embodiments described below are only for explaining the present invention, and should not be construed as limiting the present invention. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and not in idealized or overly formal meanings unless defined herein to explain.

The invention provides a reflection separation method of a light field image, which can separate a background image and a reflection image after acquiring a light field image with reflection. The reflection separation method utilizes the rich angular space information of the light field image to automatically and accurately separate the background image and the reflection image, wherein the reflection image refers to the interference image formed by reflection. Compared with the existing technology, it has the advantages of excellent separation effect and fast processing speed.

Below in conjunction with the accompanying drawings, the present invention will be further described in detail through specific embodiments.

This embodiment provides a reflection separation method for a light field image, which separates an input light field image with reflection, and outputs a background image and a reflection image at a central viewing angle.

FIG. 1 is a flow chart of a reflection separation method for a light field image in this embodiment. As shown in FIG. 1 and FIG. 3 , the reflection separation method of the light field image in this embodiment includes the following steps:

In step S1, the original light field image is obtained, and multi-scale spatial angle convolution is performed to obtain multi-scale spatial angle features.

In this step, the multi-scale spatial angle convolution is performed on the light field image, which specifically includes: first, using a spatial 2D convolution with a convolution kernel size of 3*3, a stride of 2, and a padding of 1 to perform downsampling operations, respectively using 1 and 2 times to obtain images of 1/2 original resolution and 1/4 original resolution; Four angular 2D convolutions to obtain three features of different scales. The convolution kernel size of the four spatial 2D convolutions and the first two angular 2D convolutions is 3*3, the stride is 1, and the padding is 1. The kernel size of the last two angular 2D convolutions is 3*3, the stride is 1, and the padding is 0. The spatial convolution is to convolve the above image in the x, y dimensions, and the angular convolution is to convolve the above image in the u, v dimensions. In this embodiment, the convolution operation can be expressed by the following formula:

是卷积操作，W表示权重矩阵，B表示偏置矩阵，X是输入， Y是输出，h是激活函数。优选地，在本实施例中，所有的激活函数都采用ReLU 函数，可以表示为：In formula (1),

is the convolution operation, W is the weight matrix, B is the bias matrix, X is the input, Y is the output, and h is the activation function. Preferably, in this embodiment, all activation functions adopt the ReLU function, which can be expressed as:

h(x)=max(0,x) (2)

After that, the features of different scales are fused into multi-scale spatial angle features by utilizing 1 spatial 2D convolution. The scale here is the resolution; the size of the convolution kernel of the spatial 2D convolution is 3*3, the stride is 1, and the padding is 1.

It should be noted that 2D convolution includes spatial 2D convolution and angular 2D convolution. When there are no restrictions on the two convolution methods, the description of 2D convolution is used, and spatial 2D convolution and/or angular 2D convolution are performed as needed. product.

Step S2, by performing a refocusing operation on the original light field image, a refocusing image is obtained.

In this step, the refocusing operation can be expressed as:

In formula (3), d represents the value of the refocusing operation, L represents the light field image, u, v represent the angular dimension, x, y represent the spatial dimension, u ^c , v ^c represent the angular coordinates of the central viewing angle, u∈[0 ,U], v∈[0,V], U and V are two constants, and I _d (x, y) represents the refocusing image after the refocusing operation using numerical values. FIG. 2 is a diagram showing an example of generation of a refocusing image.

After the above-mentioned refocusing operation, the acquired refocusing images are generally multiple. Preferably, in this embodiment, d∈[-2,2], the step size is 0.4, and a total of 11 refocusing images are generated.

In step S3, a dynamic convolution operation is performed on the refocusing image, and a preliminary background image and a preliminary reflection image are respectively generated.

In this step, two different dynamic convolutions are performed on the refocusing image, which is realized by 5 serial 3D convolutions, and a background dynamic convolution kernel and a reflection dynamic convolution kernel are respectively generated. A preliminary background image is generated from the refocused image, and a preliminary reflected image is generated from the refocused image using a reflection dynamic convolution kernel. The specific formula is as follows:

B _inter (x,y)＝∑ _d I(d,x,y)*W _B (d,x,y) (4)

R _inter (x,y)＝∑ _d I(d,x,y)*W _R (d,x,y) (5)

In equations (4) and (5), d represents the value of the refocusing operation, x, y represent the spatial dimension, I(d, x, y) represents all refocusing images, and B _inter (x, y) represents the preliminary background Image, WB (d,x,y) represents the background dynamic convolution kernel, R _inter (x,y) represents the preliminary reflection image, and W _{R (} d,x,y) represents the reflection dynamic convolution kernel.

Step S4: Convolve the preliminary background image and the preliminary reflection image respectively to obtain a background edge map, a background refocusing feature, a reflection edge map, and a reflection refocusing feature.

In this step, perform three spatial 2D convolutions on the preliminary background image to obtain background edge maps and background refocusing features, and perform three spatial 2D convolutions on the preliminary reflected images to obtain reflected edge maps and reflection refocusing feature. In order to better obtain the edge map, the L1 loss of the edge map is introduced for supervision.

Preferably, the size of the convolution kernel of the spatial 2D convolution in this step is 3*3, the step size is 1, and the padding is 1.

Step S5, based on the spatial attention mechanism, use the background edge map to generate background spatial attention weights, and use the reflected edge map to generate reflective spatial attention weights.

In this step, the background edge map is input into a spatial 2D convolution, and its channel is changed to 1 to obtain the background spatial attention weight; at the same time, the reflected edge map is input into a spatial 2D convolution, and its channel is becomes 1 to get the reflective spatial attention weight.

Preferably, the size of the convolution kernel of the spatial 2D convolution in this step is 3*3, the step size is 1, and the padding is 1.

Step S6, combining the multi-scale space angle feature and the background refocusing feature to obtain the background reconstruction feature, and combining the multi-scale space angle feature and the reflection refocusing feature to obtain the reflection reconstruction feature.

In this step, the multi-scale spatial angle feature and the background refocusing feature are input into one spatial 2D convolution to obtain the background reconstruction feature; at the same time, the multi-scale spatial angle feature and the reflection refocusing feature are input into another spatial 2D convolution In the convolution, the reflection reconstruction features are obtained.

Preferably, the size of the convolution kernel of the spatial 2D convolution in this step is 3*3, the step size is 1, and the padding is 1.

In step S7, based on the channel attention mechanism, the background channel attention weight is generated by using the background reconstruction feature, and the reflection channel attention weight is generated by using the reflection reconstruction feature.

In this step, the global maximum pooling operation and the global average pooling operation are performed on the background reconstruction feature, and its spatial resolution becomes 1*1, the background maximum pooling feature and the background average pooling feature are obtained, and the background maximum pooling feature and the background average pooling feature are obtained. The pooling feature and background average pooling feature are input into the fully connected layer to obtain the attention weight of the background channel; similarly, the global maximum pooling operation and the global average pooling operation are performed on the reflection reconstruction feature to change its spatial resolution to 1 *1. Obtain the reflection maximum pooling feature and the reflection average pooling feature, and input the reflection maximum pooling feature and the reflection average pooling feature into the fully connected layer to obtain the reflection channel attention weight.

Step S8, utilize the background space attention weight and the background channel attention weight to adjust the background reconstruction feature, and obtain the adjusted background reconstruction feature; use the reflection space attention weight and the reflection channel attention weight to adjust the reflection Reconstruct the feature to obtain the adjusted reflection reconstruction feature.

In this step, the adjusted reconstruction features are obtained according to formulas (6) and (7):

In equations (4) and (5), x, y represent the spatial dimension, CA _B (x, y) represents the background channel attention weight,

表示调整后的背景重构特征；CA_R(x,y)表示反射通道注意力权重，SA_R(x,y)表示反射空间注意力权重，F_R(x,y)表示反射重构特征，

表示调整后的反射重构特征。SA _B (x, y) represents the background spatial attention weight, F _B (x, y) represents the background reconstruction feature,

Represents the adjusted background reconstruction features; _CAR (x,y) represents the reflection channel attention weight, SAR (x,y) represents the reflection spatial attention weight, F _{R (} x,y) represents the reflection reconstruction feature,

Represents the adjusted reflection reconstruction feature.

In step S9, the background image is reconstructed from the adjusted background reconstruction feature by using a U-shaped network, and the reflection image is reconstructed from the adjusted reflection reconstruction feature by using the U-shaped network.

In this step, the background reconstruction feature and the reflection reconstruction feature are respectively input to two different U-shaped networks. The U-shaped network includes an encoding layer, a decoding layer and a skip layer connection. The encoding layer consists of three groups including spatial 2D convolution The spatial 2D convolution here includes two steps of 1 and 2; the decoding layer is composed of 3 groups of modules including spatial 2D convolution and transposed convolution; the skip layer structure is used to connect the encoding layer and decoding layer. L1 loss supervision is added to the background image and reflection image during training; finally, the background image and reflection image are obtained from the U-shaped network. When using the U-shaped network to obtain the background image and the reflection image, the encoding layer performs 2 downsampling operations (spatial 2D convolution with kernel size 3*3, stride 2, and padding 1) and 3 spatial convolutions. 2D convolution (the kernel size is 3*3, the stride is 1, and the padding is 1); the decoding layer performs 2 upsampling operations (transposed convolution) and 3 spatial 2D convolutions (the kernel size is 1). 3*3, step size is 1, padding is 1).

It can be seen from the above that the reflection separation method of the light field image according to the embodiment of the present invention can automatically obtain a background image and a reflection image only through an original light field image with reflection; the separation effect is good. , the processing speed is fast, the robustness to real complex scenes is high, the obtained background images and reflection images are independent and rich in details, and can be widely used in common computer vision tasks to improve their processing performance, such as: target detection , target tracking, three-dimensional reconstruction, etc. For example, the separated background image can be used for more accurate target detection and tracking, and the separated reflection image can be used for the detection of cases in the criminal investigation field.

Based on the same idea, an embodiment of the present invention also provides a light field image reflection separation system, the system includes: a light field image acquisition module, a multi-scale spatial angle feature generation module, a refocusing module, and a preliminary separation image generation module , edge map generation module, spatial attention weight generation module, refocusing feature generation module, reconstruction feature generation module, channel attention weight generation module, reconstruction feature adjustment module, image separation module.

Wherein, the light field image acquisition module is used to acquire the original light field image;

The multi-scale spatial angle feature generation module is used to perform multi-scale spatial angle convolution on the original light field image, obtain multi-scale spatial angle features, and send them to the reconstructed feature generation module;

The refocusing module is used for performing a refocusing operation on the original light field image to obtain the refocusing image;

The preliminary separated image generation module is used to perform a dynamic convolution operation on the refocusing image to generate a preliminary background image and a preliminary reflection image respectively;

The edge map generation module is used to convolve the preliminary background image and the preliminary reflected image respectively to obtain the background edge map and the reflected edge map;

The spatial attention weight generation module is configured to generate background spatial attention weights by using the background edge map based on the spatial attention mechanism, and generate reflective spatial attention weights by using the reflected edge map;

The refocusing feature generation module is configured to convolve the preliminary background image and the preliminary reflection image respectively to obtain the background refocusing feature and the reflection refocusing feature;

The reconstruction feature generation module is used for combining the multi-scale spatial angle feature and the background refocusing feature to obtain the background reconstruction feature, and combining the multi-scale spatial angle feature and the reflection refocusing feature to obtain the reflection reconstruction feature;

The channel attention weight generation module is used to generate the background channel attention weight by using the background reconstruction feature, and use the reflection reconstruction feature to generate the reflection channel attention weight;

The reconstruction feature adjustment module is used to adjust the background reconstruction feature by using the background space attention weight and the background channel attention weight to obtain the adjusted background reconstruction feature; using the reflection space attention weight and reflection The channel attention weight adjusts the reflection reconstruction feature, and obtains the adjusted reflection reconstruction feature;

The image separation module is used for reconstructing the background image from the adjusted background reconstruction features by using the U-shaped network, and reconstructing the reflection image from the adjusted reflection reconstruction features by using the U-shaped network.

In this embodiment, each module is implemented by a processor, and memory is appropriately added when storage is required. The processor may be, but not limited to, a microprocessor MPU, a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), and an application-specific integrated circuit (ASIC). ), field programmable gate arrays (FPGA), other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, etc. The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.).

It should also be noted that the light field image reflection separation system in this embodiment corresponds to the reflection separation method, and the description and limitations of the method are also applicable to the system, and are not repeated here.

The above descriptions are the preferred embodiments of the present invention. It should be noted that the present invention is not limited to the exemplary embodiments disclosed above, and the essence of the description is only to help those skilled in the relevant art to comprehensively understand the specific details of the present invention. For those of ordinary skill in the art, without departing from the principles of the present invention, several improvements and modifications, and easily conceivable changes or substitutions made within the technical scope of the present invention should be covered in the within the protection scope of the present invention.

Claims (10)

1. a reflection separation method of light field image, is characterized in that, described reflection separation method comprises the steps: Step S1, obtain the original light field image, perform multi-scale spatial angle convolution, and obtain multi-scale spatial angle features; Step S2, obtaining a refocusing image by performing a refocusing operation on the original light field image; Step S3, performing a dynamic convolution operation on the refocusing image to generate a preliminary background image and a preliminary reflection image respectively; Step S4, respectively convolving the preliminary background image and the preliminary reflection image to obtain a background edge map, a background refocusing feature, a reflection edge map, and a reflection refocusing feature; Step S5, based on the spatial attention mechanism, using the background edge map to generate background spatial attention weights, and using the reflected edge map to generate reflective spatial attention weights; Step S6, merging the multi-scale space angle feature and the background refocusing feature to obtain the background reconstruction feature, and merging the multi-scale space angle feature and the reflection refocusing feature to obtain the reflection reconstruction feature; Step S7, based on the channel attention mechanism, utilize the background reconstruction feature to generate the background channel attention weight, and utilize the reflection reconstruction feature to generate the reflection channel attention weight; Step S8, utilize the background space attention weight and the background channel attention weight to adjust the background reconstruction feature, and obtain the adjusted background reconstruction feature; use the reflection space attention weight and the reflection channel attention weight to adjust the reflection Reconstruction features to obtain adjusted reflection reconstruction features; In step S9, the background image is reconstructed from the adjusted background reconstruction feature by using a U-shaped network, and the reflection image is reconstructed from the adjusted reflection reconstruction feature by using the U-shaped network. 2. The light field image reflection separation method according to claim 1, wherein the step S1 multi-scale spatial angle convolution specifically comprises the following steps: Use spatial 2D convolution to perform downsampling operations, using 1 and 2 times respectively, to obtain images of 1/2 original resolution and 1/4 original resolution; Perform 4 spatial 2D convolutions and 4 angular 2D convolutions on the original, 1/2 original resolution and 1/4 original resolution images respectively to obtain features of different scales; The features of different scales are fused by spatial 2D convolution to obtain multi-scale spatial angle features. 3. The light field image reflection separation method according to claim 1, wherein the formula of the refocusing operation in the step S2 is:

In formula (1), d represents the value of the refocusing operation, L represents the light field image, u, v represent the angular dimension, x, y represent the spatial dimension, u ^c , v ^c represent the angular coordinates of the central viewing angle, u ∈ [0 , U], v∈[0, V], U and V are two constants, and I _d (x, y) represents the refocusing image after the refocusing operation using numerical values. 4. The light field image reflection separation method according to claim 1, wherein the step S3 dynamic convolution specifically comprises the following steps: Input the refocused image into 2 different 3D convolutions to generate background dynamic convolution kernel and reflection dynamic convolution kernel respectively; generating a preliminary background image from the refocused image using a background dynamic convolution kernel; A preliminary reflection image is generated from the refocused image using a reflection dynamic convolution kernel. 5. light field image reflection separation method according to claim 4, is characterized in that, described step S3 utilizes dynamic convolution to generate the operation of preliminary background image and preliminary reflection image, adopts following formula to express: B _inter (x, y) = ∑ _d I(d, x, y)*W _B (d, x, y) (2) R _inter (x, y) = ∑ _d I(d, x, y)*W _R (d, x, y) (3) In equations (2) and (3), d represents the value of the refocusing operation, x, y represent the spatial dimension, I(d, x, y) represents all refocusing images, and B _inter (x, y) represents the preliminary background image, W _B (d, x, y) represents the background dynamic convolution kernel, R _inter (x, y) represents the preliminary reflection image, and W _R (d, x, y) represents the reflection dynamic convolution kernel. 6 . The light field image reflection separation method according to claim 1 , wherein the step S5 is based on a spatial attention mechanism, using the background edge map to generate background spatial attention weights, and using the reflection edge map to generate Reflecting the spatial attention weight, which includes the following steps: Input the background edge map into the spatial 2D convolution, change its channel to 1, and obtain the background spatial attention weight; The reflection edge map is input into the spatial 2D convolution, and its channel is changed to 1 to obtain the reflection spatial attention weight. 7. The light field image reflection separation method according to claim 1, wherein the step S7 is based on a channel attention mechanism, using the background reconstruction feature to generate a background channel attention weight, and using the reflection Reconstructing features to generate reflection channel attention weights includes the following steps: Perform global maximum pooling and global average pooling operations on the background reconstruction feature, change its spatial resolution to 1*1, and obtain background maximum pooling features and background average pooling features; Input the background maximum pooling feature and background average pooling feature into the fully connected layer to obtain the background channel attention weight; Perform global maximum pooling and global average pooling operations on the reflection reconstruction feature, change its spatial resolution to 1*1, and obtain reflection maximum pooling features and reflection average pooling features; Input the reflection max pooling feature and reflection average pooling feature into the fully connected layer to obtain the reflection channel attention weight. 8. The light field image reflection separation method according to claim 1, wherein the step S8 obtains the adjusted background reconstruction feature and the reflection reconstruction feature, and obtains by the following formula:

In equations (4) and (5), x, y represent the spatial dimension, CA _B (x, y) represents the background channel attention weight, SA _B (x, y) represents the background spatial attention weight, F _B (x, y) represents the background reconstruction feature,

Represents the adjusted background reconstruction features; _CAR (x, y) represents the reflection channel attention weight, SAR (x, y) represents the reflection spatial attention weight, F _{R (} x, y) represents the reflection reconstruction feature,

Represents the adjusted reflection reconstruction feature. 9. The light field image reflection separation method according to claim 1, wherein the structure of the U-shaped network in the step S9 specifically includes an encoding layer, a decoding layer and a layer-hopping structure; wherein, The coding layer is composed of 3 groups of modules including spatial 2D convolution; The decoding layer is composed of 3 groups of modules including spatial 2D convolution and transposed convolution; The skip layer structure is used to connect the coding layer and the decoding layer. 10. A reflection separation system for light field images, characterized in that the system comprises: a light field image acquisition module, a multi-scale spatial angle feature generation module, a refocusing module, a preliminary separation image generation module, an edge map generation module, Spatial attention weight generation module, refocusing feature generation module, reconstruction feature generation module, channel attention weight generation module, reconstruction feature adjustment module, and image separation module; among them, The light field image acquisition module is used to acquire the original light field image; The multi-scale spatial angle feature generation module is used to perform multi-scale spatial angle convolution on the original light field image, obtain multi-scale spatial angle features, and send them to the reconstructed feature generation module; The refocusing module is used to perform a refocusing operation on the original light field image to obtain the refocusing image; The preliminary separated image generation module is used to perform a dynamic convolution operation on the refocusing image to generate a preliminary background image and a preliminary reflection image respectively; The edge map generation module is used to convolve the preliminary background image and the preliminary reflected image respectively to obtain the background edge map and the reflected edge map; The spatial attention weight generation module is configured to generate background spatial attention weights by using the background edge map based on the spatial attention mechanism, and generate reflective spatial attention weights by using the reflected edge map; The refocusing feature generation module is configured to convolve the preliminary background image and the preliminary reflection image respectively to obtain the background refocusing feature and the reflection refocusing feature; The reconstruction feature generation module is used for combining the multi-scale spatial angle feature and the background refocusing feature to obtain the background reconstruction feature, and combining the multi-scale spatial angle feature and the reflection refocusing feature to obtain the reflection reconstruction feature; The channel attention weight generation module is used to generate the background channel attention weight by using the background reconstruction feature, and use the reflection reconstruction feature to generate the reflection channel attention weight; The reconstruction feature adjustment module is used to adjust the background reconstruction feature by using the background space attention weight and the background channel attention weight to obtain the adjusted background reconstruction feature; using the reflection space attention weight and reflection The channel attention weight adjusts the reflection reconstruction feature, and obtains the adjusted reflection reconstruction feature; The image separation module is used for reconstructing the background image from the adjusted background reconstruction features by using the U-shaped network, and reconstructing the reflection image from the adjusted reflection reconstruction features by using the U-shaped network.

Images