CN114967121B - An end-to-end single-lens imaging system design method - Google Patents

Abstract

The invention discloses a design method of an end-to-end single lens imaging system, which comprises the following steps: constructing a single-lens imaging system frame; calculating the square error loss and the additional constraint loss of the restored image and the original image, and constructing a loss function based on the square error loss and the additional constraint loss; and iteratively optimizing trainable parameters of the single-lens imaging system framework to construct the single-lens imaging system. The invention optimally designs the single-lens imaging system, so that the single-lens imaging system has a better initial structure, the training difficulty of deep learning is greatly reduced, and certain performance requirements can be met.

Description

An end-to-end single-lens imaging system design method

technical field

The invention belongs to the technical field of computational optical imaging, in particular to an end-to-end single-lens imaging system design method.

Background technique

Compared with complex lens groups, single lenses have the advantages of small size, light weight and simple structure. However, the aberration of ordinary single lenses is large, especially when imaging with a large field of view, the obtained images are often blurred. To achieve high-quality optical system imaging, it is often necessary to use complex lens combinations to correct aberrations. However, the complex lens group optical system has the disadvantages of large volume, high mass and high cost, which has certain limitations in applications that require miniaturization, such as mobile phone lenses, drone platform camera systems, and remote sensing cameras.

The single-lens imaging system acquires images through a single lens, and uses post-processing algorithms to correct the blur caused by the single-lens aberration in the image. There is an urgent need for miniaturized imaging systems such as cameras. Single-lens imaging systems can be divided into discrete design systems and end-to-end design systems. The discrete design system first designs the single lens, and then designs the post-processing restoration algorithm according to the imaging effect of the single lens. The end-to-end single-lens imaging system design connects the imaging simulation and restoration algorithm, and uses deep learning technology to simultaneously learn and train the single-lens surface parameters and restoration algorithm parameters. Compared with discrete design, end-to-end design has the advantage of global optimization.

In the existing end-to-end single-lens design method, the restoration network part uses a convolutional neural network, which cannot learn position information, and has a poor restoration effect on aberrations related to spatial positions. Moreover, these existing methods lack constraints on additional boundary conditions such as single lens edge thickness, center thickness, and energy distribution, which will cause the optical lens designed by the algorithm to fail to meet the requirements of practical processing applications.

Contents of the invention

The purpose of the present invention is to provide an end-to-end single-lens imaging system design method to solve the above-mentioned problems in the prior art.

To achieve the above object, the present invention provides an end-to-end single-lens imaging system design method, including:

Calculate the square difference loss and additional constraint loss of the restored image and the original image, and construct a loss function based on the square difference loss and the additional constraint loss;

A single-lens imaging system framework is constructed, and based on deep learning and the loss function, the single-lens imaging system framework is iteratively optimized to obtain an optimized system, and the optimized system is used as a single-lens imaging system.

Optionally, the single-lens imaging system framework includes: an optical image blur simulation module, a blur kernel learning module and an inverse filter image restoration module;

The process of building the framework of a single-lens imaging system includes:

Establish the mapping equation of the point spread function of each field of view and each waveband of the single lens with respect to its surface shape parameters, perform convolution interpolation based on the point spread function and the original image, obtain a single lens blur simulation map, and construct the optical image blur simulation module;

Using the point spread function of the 0° field of view in the point spread function as an estimated blur kernel, constructing a neural network, correcting the estimated blur kernel based on the neural network, obtaining a correction result, and constructing the fuzzy kernel learning module;

Constructing the inverse filter image restoration module based on the adaptive Wiener filtering method and the fuzzy kernel output by the fuzzy kernel learning module;

The single-lens imaging system framework is constructed based on the optical image blur simulation module, the blur kernel learning module and the inverse filter image restoration module.

Optionally, in the process of constructing the optical image fuzzy simulation module, the point spread function is obtained based on the principle of geometric optics and ray tracing; the aspherical parameters of the single lens are set as surface parameters, and the single lens The blur simulation map is differentiable with respect to the aspheric parameters.

Optionally, the neural network is a three-layer fully connected neural network with a skip connection structure, and each fully connected layer includes 27×27 neurons.

Optionally, the method for the neural network to modify the estimated blur kernel is: transform the estimated blur kernel into a one-dimensional matrix form and input it into the neural network; calculate the output of each layer of neural network As a result, the output of the third layer is transformed into the form of an image matrix, forming a corrected blur kernel and zero-padding.

Optionally, the algorithmic expression of the adaptive Wiener filtering method is:

表示复原图像，F(·)表示傅里叶变换，F^-1(·)表示傅里叶逆变换，K为通过学习训练自适应调整的可优化的参数，I₁为单透镜模糊仿真图像，

为模糊核学习模块输出的模糊核。in,

Represents the restored image, F( ) represents the Fourier transform, F ^-1 ( ) represents the inverse Fourier transform, K is the parameter that can be optimized through learning and training adaptive adjustment, I ₁ is the single-lens blur simulation image,

The fuzzy kernel output by the fuzzy kernel learning module.

Optionally, the formula for calculating the square difference loss mseloss between the restored image and the original image is:

Among them, m, n represent the size of the image, i, j represent the pixel position in the image, I ₀ represents the original image, mseloss represents the square difference loss between the original image and the restored image.

Optionally, the calculation method of the additional constraint loss is:

In the formula, loss ₀ represents the additional constraint loss, k _n represents the current value of the variable, k _y represents the variable threshold value, and sigmoid represents the activation function. When k _n ≥ k _y , the additional loss value is inactive, and when k _n < k _y is activated Additional loss value.

Optionally, the method of constructing a loss function based on the square difference loss and the additional constraint loss is: performing weighted summation of the square difference loss and the additional constraint loss, and constructing a loss function based on a result of the weighted summation.

Optionally, the method for iteratively optimizing the frame of the single-lens imaging system is:

The initial aspheric parameters and neural network parameters are initially set to 0, and the aspheric parameters, neural network and inverse filter adaptive parameters are jointly optimized.

The present invention has the following technical effects:

(1) The present invention has set up a kind of end-to-end single-lens imaging system frame, according to the imaging effect of the system simultaneously in the optical system surface shape parameter in the single-lens imaging system frame, ResDNN3 neural network parameter and Wiener filtering image restoration algorithm The noise constant parameters are optimized.

(2) The present invention proposes a fully-connected neural network (ResDNN3) with a skip connection structure. This network takes the estimated blur kernel as input and can be used for learning and correcting the blur kernel of the optical system.

(3) The present invention adds an additional constraint loss of the optical system in the training optimization of the end-to-end single-lens imaging system, which can constrain the edge thickness and energy distribution of the designed single-lens.

(4) The present invention proposes a set of initialization methods for designing the single-lens imaging system framework, so that the single-lens imaging system of the present invention has a better initial structure, and this system framework has a good initial value during training, greatly reducing the need for The difficulty of training optimization for this system framework.

Description of drawings

The drawings constituting a part of the application are used to provide further understanding of the application, and the schematic embodiments and descriptions of the application are used to explain the application, and do not constitute an improper limitation to the application. In the attached picture:

FIG. 1 is a flow chart of an end-to-end single-lens imaging system design method in an embodiment of the present invention;

Fig. 2 is the ResDNN3 network structure diagram in the embodiment of the present invention;

Fig. 3 is the optical path diagram of single lens in the embodiment of the present invention;

Fig. 4 is the comparison diagram of the point spread function of each field of view of the single lens before and after optimization in the embodiment of the present invention, wherein (a) is the point spread function of each field of view of the unoptimized optical system, (b) is the optical obtained by learning optimization Point spread function of each field of view of the system;

Fig. 5 is the learning optimization effect of the fuzzy kernel in the embodiment of the present invention, wherein (a) is an unoptimized estimated fuzzy kernel, (b) is a fuzzy kernel after learning optimization;

Detailed ways

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be performed in a computer system, such as a set of computer-executable instructions, and that although a logical order is shown in the flowcharts, in some cases, The steps shown or described may be performed in an order different than here.

Embodiment one

This embodiment proposes an end-to-end single-lens imaging system design method, which can be effectively applied to areas such as smartphone cameras, drone platform camera systems, and remote sensing cameras that urgently need miniaturized imaging systems. The flow chart is shown in Figure 1 shown. In this embodiment, a single-lens imaging system with a focal length of 43.5mm, a clear aperture of 23.4mm, and a full field of view of 47° is taken as an example to introduce specific implementation methods of the present invention:

Step 1: The established end-to-end single-lens imaging system framework includes three modules: optical image blur simulation module, blur kernel learning module and inverse filter image restoration module. The establishment process of these three modules is as follows:

Step 1-1, establishing an optical image blurring simulation module. Firstly, the mapping equation of the point spread function of each field of view and each waveband of the single lens with respect to its surface shape parameters is established, and then the point spread function is used to perform convolution interpolation with the original image to obtain a single lens blur simulation image to realize optical image blur simulation.

The aspheric parameters of the single lens are set as the surface parameters that can be optimized, and the blurred image in the established image simulation model is differentiable with respect to the aspheric parameters. The single lens used in the present invention is a plano-convex even-order aspheric lens, the front surface of the lens is a plane, and the rear surface is a curved surface using the 4th, 6th, 8th, and 10th-order even-order aspheric parameters.

Firstly, based on the principle of geometric optics, the present invention establishes the mapping equation of the incident position and direction of the light under the single-lens optical system to the position of the image plane through ray tracing to establish the functional relationship between the surface shape parameters and the point spread function:

G(x ₀ ,y ₀ ,θ,{a ₄ ,a ₆ ,a ₈ ,a ₁₀ })=(x ₁ ,y ₁ )

Where x ₀ , y ₀ , θ are the incident position and direction, {a _n } is the set of surface shape parameters to be optimized, x ₁ , y ₁ are the position of the image plane, G( ) is the incident position and direction of light to the image The mapping transformation of the face position. To establish this mapping equation, the surface shape function F(x, y, z, a _n )=0 of the optical lens, the refractive index of the lens material and the position of each optical surface need to be known. The general surface shape equation for an even-order aspheric surface is:

Among them, r is the position in the radial direction, z is the vector radius at the corresponding position, c is the curvature at the apex, r is the radius, k is the conicity, and a ₂ , a ₄ , a ₆ etc. are aspheric coefficients. In the example of the present invention, the lens thickness d is 6 mm, the vertex curvature radius r is -21.4 mm, the conicity k is 0, and a ₄ , a ₆ , a ₈ and a ₁₀ are the aspheric coefficients to be optimized.

The lens material used in the example is PMMA, and its refractive index to the central wavelength of the visible light band is n ₁ =1.49. Using the law of spatial refraction of light, the azimuth angle after light refraction can be obtained by using the law of space refraction and the law of linear propagation of light. The spatial position and direction angle of each refraction of light on the lens surface can be solved layer by layer to obtain x ₁ , y ₁ from x ₀ , y ₀ , θ.

Uniformly select 7×7 point diffusion sampling areas on the image plane, and the size of each sampling area is 27×27 pixels, and calculate the incident angle of the chief ray corresponding to the center position of these point diffusion sampling areas. For each incident angle, create 64×64 parallel sampling ray arrays on the incident surface, and calculate the distribution of these rays in the point-diffusion sampling area, and the amount of light in each pixel block in the point-diffusion sampling area is taken as the point The parameter value of the corresponding parameter in the spread function.

Assuming that the spatial position of each pixel center in the diffuse sampling area of each point on the image plane is x ₂ , y ₂ , then the distance between the center of each pixel on the image plane and the falling point of light is:

Then, the Gaussian function is used to diffuse the light falling point of the image plane, and different weights are assigned to the pixels according to the distance between the light falling point and the center of the pixel space:

Among them, the standard deviation of the σ Gaussian function can be adjusted according to the pixel size of the detector. The point spread function of the corresponding position can be obtained by superimposing and normalizing the intensity distribution of all the sampling ray landing points at the same incident angle:

Use these point spread functions to convolve with the original image, and then use the interpolation function to fuse the degraded images convolved with the point spread functions into a fuzzy simulation image and add noise terms. This process is described as:

Among them, I ₀ is the original image, I ₁ is the single-lens blur simulation image, η is the noise, and SINC _ij is the weight map of the SINC function with the center point at (i,j) position.

Step 1-2, establishing a fuzzy kernel optimization learning module. Use the point spread function of the central field of view obtained in step 1-1 as the estimated blur kernel, and establish a neural network to transform the estimated blur kernel into a blur kernel that characterizes the characteristics of the entire optical system, thereby realizing the function of learning the blur kernel.

The estimated blur kernel psf ₀₀ used is the point spread function of the central band of the central field of view obtained in step 1, which is transformed into a one-dimensional matrix H ₀ =[h ₀ ,h ₁ ,…,h _27×27 ] , input into the ResDNN3 network established by the present invention.

The structure of the ResDNN3 network established in this paper is shown in Figure 2. It is a three-layer fully connected neural network with a skip connection structure. All neurons in the fully connected neural network are connected to each other, and each fully connected layer has 27×27 neurons. Skip connection means that the output of each layer of the neural network must be added to its input to enrich the network information.

表示第x层全连接网络中的第j个神经元，

表示上一层输入中的第i项，

表示上一层中第j个神经元，

和

分别表示此条连接的权重和偏置，则该神经网络中每个神经元的计算方式为：The way of skipping the connection can introduce the information of the neurons of the previous layer in a simple way, and it hardly increases the amount of additional calculation. At present, this method is widely used in the convolutional neural network, and it can also play a role in the fully connected neural network. produce the corresponding effect. make

Represents the jth neuron in the fully connected network of the xth layer,

Represents the i-th item in the input of the previous layer,

Indicates the jth neuron in the previous layer,

and

represent the weight and bias of this connection respectively, then the calculation method of each neuron in the neural network is:

Among them, the weight w _i and bias b _i are used as the optimization variables of the neural network, and the calculation results of all neurons in each layer of fully connected neural network are used as the output of this layer of neural network. Thus, the output results of each layer of the neural network in the present invention are in turn:

表示。Transform the output result H ₃ of the third layer into the form of image matrix to form the corrected blur kernel psf _H3 . Pad the corrected blur kernel to the same size as the blurred image I ₁ , and use the point spread function after zero padding

express.

Steps 1-3, establishing an inverse filter image restoration module. The adaptive Wiener filtering method is used as the inverse filtering algorithm of the image restoration module, and the blur kernel used by the image restoration module is the blur kernel output by the fuzzy kernel learning module.

表示复原图像，S_η为噪声的功率谱，S_f为退化函数的功率谱，F(·)表示傅里叶变换，

为模糊核优化学习模块输出的模糊核，于是利用维纳滤波复原图像的算法表达式为：Wiener filtering, also known as minimum mean square error filtering, is an inverse filtering restoration algorithm that considers noise interference. make

Represents the restored image, S _η is the power spectrum of the noise, S _f is the power spectrum of the degradation function, F(·) represents the Fourier transform,

The fuzzy kernel output from the learning module is optimized for the fuzzy kernel, so the algorithm expression for image restoration using Wiener filtering is:

项难以被精确计算，以往使用时通常将其设为常量。本发明中将其设为可优化的参数，通过学习训练自适应调整。设F^-1(·)为傅里叶逆变换，K值为自适应参数，则复原图像

的表达式为：in,

The term is difficult to be calculated precisely, and it is usually set as a constant when used in the past. In the present invention, it is set as an optimizable parameter, which is adaptively adjusted through learning and training. Let F ^-1 ( ) be the inverse Fourier transform, K is an adaptive parameter, then restore the image

The expression is:

In said step 2, let m, n represent the size of the image, i, j represent the pixel position in the image, and mseloss represent the square difference loss between the original image and the restored image, then the square difference loss between the original image and the restored image is:

The additional constraint loss of the optical system is introduced by the difference between the current value of the variable k _n and the set variable threshold k _y , and the sigmoid function is used as the activation function. When k _n ≥ k _y , this loss item is not activated. When k _n < k _y Activate the additional loss value when . Specifically, the present invention can impose additional constraints on the lens edge thickness and energy distribution, and its general paradigm is:

Among them, for a single lens, the variable current value k _n mainly has two types: lens edge thickness and energy distribution. Suppose r is the position in the radial direction, the radii R ₁ (r) and R ₂ (r) are the coordinate functions of the front surface and the rear surface in the direction of the optical axis respectively, and r ₀ is the radial radius of the lens, then the edge thickness The expression of variable current value k _n1 is:

k _n1 =R ₂ (r ₀ )-R ₁ (r ₀ )

The energy value of the system can be expressed by the sum of the above-mentioned sampled parallel light arrays on the image plane, the sum of the energy within the point diffusion sampling range, or the energy sum of the rays in a smaller range or even in the central pixel. Let m and n are the range of energy transfer under investigation, then the current value of the variable of energy distribution is:

k _n2 = sum(psf(x,y)),x≤m,y≤n

The weighted sum of the square difference loss and the additional constraint loss is obtained to obtain the loss function loss of the entire end-to-end design, and the weights of the square difference loss and the additional constraint loss are respectively α and β, then:

loss＝α*mseloss+β ₁ *loss ₀ (k _n1 )+β ₂ *loss ₀ (k _n2 )

Step 3: Use deep learning technology to iteratively optimize the trainable parameters of the end-to-end single-lens imaging system to obtain the optimal single-lens imaging system parameters

Initialize the initial aspheric parameters and resDNN3 network parameters to 0, so in the initial training of deep learning: the optical system is a standard spherical mirror, and the estimated blur kernel is the point spread function of the standard spherical mirror in the 0° field of view. The calculation result of the estimated blur kernel is still the point spread function of the standard spherical mirror in the 0° field of view, and the initial value of the noise characteristic parameter of the Wiener filter is set to 0.01. Compared with random initialization parameters, the initialization method of the present invention has a certain physical meaning, and better imaging effects can be obtained when the system is first trained. Therefore, the single-lens imaging system of the present invention has a better initial structure, which greatly reduces the training difficulty of deep learning.

A clear scene image with an image size of 256×256 is selected as the data set, in which there are 350 pictures in the training set and 50 pictures in the test set. In each iteration of deep learning training, the loss value of the training set is calculated for gradient descent, and the parameters are updated; the loss value of the test set is calculated to verify the effect of the single-lens imaging system, and the parameters are not updated. Set the learning rate to 1×10 ^-4 , repeat iterations 100 times, and select the primary parameter with the smallest loss value of the test set as the final optimization result.

The optimized values of the aspheric parameters a ₄ , a ₆ , a ₈ and a ₁₀ of the single lens are respectively 1.766×10 ^-5 , -9.100×10 ^-9 , -4.052×10 ^-11 and 9.894×10 ^-12 , The noise constant parameter K=2×10 ⁻⁴ obtained through optimization in Wiener filtering. The optical path diagram of the designed single lens is shown in Figure 3. From the comparison of the unoptimized optical system standard spherical mirror (a) and the optical system (b) with optimized aspheric parameters in Figure 4, it can be seen that the end-to-end single-lens imaging system optimizes the single-lens surface shape so that the single-lens The point spread function of the edge field of view is more concentrated and more consistent with the point spread function of the central field of view. Therefore, a good restoration effect can be obtained by using a learned blur kernel to perform inverse filter restoration on the single-lens blurred image. From the comparison of the predicted blur kernel (a) in Figure 5 and the learned blur kernel (b) that characterizes the full field of view of a single lens, it can be seen that the learned blur kernel is more complex, and its distribution is related to the spatial position. ResDNN3 The network has played a good learning effect.

The restoration algorithm part of the single-lens imaging system can restore the blurred image acquired by the single-lens to a clear image close to the original image. The peak signal-to-noise ratio and structural similarity are used as evaluation indicators to measure the similarity of the two images. After calculation, the structural similarity between the blurred image acquired by the single lens and the original image is 0.63, and the peak signal-to-noise ratio is 20.93; after restoration The structural similarity between the original image and the original image is 0.93, and the peak signal-to-noise ratio is 25.47. The comparison of quantitative indicators also proves the effectiveness of the design of the present invention.

The invention establishes an end-to-end single-lens imaging system framework, and simultaneously optimizes the surface shape parameters of the optical system, ResDNN3 neural network parameters and noise constant parameters in the Wiener filter image restoration algorithm according to the imaging effect of the system.

The present invention proposes a fully-connected neural network (ResDNN3) with a skip connection structure. The network takes the estimated blur kernel as input and can be used for learning and correcting the blur kernel of the optical system.

In the present invention, an additional constraint loss of the optical system is added to the training optimization of the end-to-end single-lens imaging system, which can constrain the edge thickness and energy distribution of the designed single-lens.

The present invention proposes a set of initialization methods for the designed single-lens imaging system frame, so that the system frame has a good initial value during training, and greatly reduces the difficulty of training and optimizing the system frame.

The above is only a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any skilled person familiar with the technical field can easily think of changes or Replacement should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (8)

1. an end-to-end single-lens imaging system design method, is characterized in that, comprises the following steps: Calculate the square difference loss and additional constraint loss of the restored image and the original image, and construct a loss function based on the square difference loss and the additional constraint loss; Constructing a single-lens imaging system framework, performing iterative optimization on the single-lens imaging system framework based on deep learning and the loss function, obtaining an optimized system, and using the optimized system as a single-lens imaging system; The single-lens imaging system framework includes: an optical image blur simulation module, a blur kernel learning module and an inverse filter image restoration module; The process of building the framework of a single-lens imaging system includes: Establish the mapping equation of the point spread function of each field of view and each waveband of the single lens with respect to its surface shape parameters, perform convolution interpolation based on the point spread function and the original image, obtain a single lens blur simulation map, and construct the optical image blur simulation module; Using the point spread function as an estimated fuzzy kernel, constructing a neural network, correcting the estimated fuzzy kernel based on the neural network, obtaining a correction result, and constructing the fuzzy kernel learning module; Constructing the inverse filter image restoration module based on the adaptive Wiener filtering method and the fuzzy kernel output by the fuzzy kernel learning module; Constructing the single-lens imaging system framework based on the optical image blur simulation module, the blur kernel learning module and the inverse filter image restoration module; The calculation method of the additional constraint loss is:

In the formula, loss ₀ represents the additional constraint loss, k _n represents the current value of the variable, k _y represents the variable threshold value, and sigmoid represents the activation function. When k _n ≥ k _y , the additional loss value is inactive, and when k _n < k _y is activated Additional loss value. 2. The end-to-end single-lens imaging system design method according to claim 1, characterized in that, in the process of constructing the optical image blur simulation module, the point spread is obtained based on the principle of geometric optics and ray tracing Function; set the aspheric parameters of the single lens as surface parameters, and the blur simulation graph of the single lens is differentiable with respect to the aspheric parameters. 3. The end-to-end single-lens imaging system design method according to claim 1, wherein the neural network is a three-layer fully connected neural network ResDNN3 with a skip connection structure, and each fully connected layer includes 27× 27 neurons. 4. The end-to-end single-lens imaging system design method according to claim 1, wherein the neural network corrects the estimated blur kernel by converting the estimated blur kernel into a Dimensional matrix form and input into the neural network; calculate the output result of each layer of neural network, transform the output result of the third layer into the form of image matrix, form a modified fuzzy kernel and perform zero padding. 5. the end-to-end single-lens imaging system design method according to claim 1, is characterized in that, the algorithmic expression of described adaptive Wiener filtering method is:

in,

Represents the restored image, F( ) represents the Fourier transform, F ^-1 ( ) represents the inverse Fourier transform, K is the parameter that can be optimized through adaptive adjustment through learning and training, I ₁ is the single-lens blur simulation image,

The fuzzy kernel output by the fuzzy kernel learning module. 6. The end-to-end single-lens imaging system design method according to claim 1, wherein the formula for calculating the square difference loss mseloss of the restored image and the original image is:

Among them, m, n represent the size of the image, i, j represent the pixel position in the image, I ₀ represents the original image, mseloss represents the square difference loss between the original image and the restored image. 7. The end-to-end single-lens imaging system design method according to claim 1, wherein the method of constructing a loss function based on the square difference loss and the additional constraint loss is: combining the square difference loss and the The above additional constraint loss is weighted and summed, and the loss function is constructed based on the result of the weighted sum. 8. The end-to-end single-lens imaging system design method according to claim 1, wherein the method for iteratively optimizing the single-lens imaging system framework is: combining the initial aspheric parameters with the initial parameters of the neural network If it is set to 0, the aspheric parameters, the neural network and the adaptive parameters of the inverse filter image restoration module are jointly optimized.

Images