CN107862655B - A Regularization-Based Alternate Minimization Method for High-resolution Image Reconstruction - Google Patents

Abstract

The invention discloses a regularization-based alternating minimization high-resolution image reconstruction method, which relates to the field of image processing, in particular to the fields of intelligent hardware and video monitoring. According to the theoretical analysis of sub-sampling the original low-resolution image sequence, the invention realizes the magnification of the rational number level, and makes up for the deficiency that the existing algorithm can only realize the magnification of the integer level. At the same time, the limitations of the existing super-resolution reconstruction algorithms for color image processing are explained, and a regular term containing color information is added to the alternate minimization super-resolution reconstruction algorithm to realize the optimal processing of color images.

Description

A Regularization-Based Alternate Minimization Method for High-resolution Image Reconstruction

technical field

The invention relates to the field of image processing, in particular to the fields of intelligent hardware and video surveillance.

Background technique

Super-resolution reconstruction is one of the core research contents in the field of computer vision. The input is a single image or a video image sequence. By obtaining the potential information in these low-resolution images, according to the algorithm model, a clear high-resolution image is finally output. image. Super-resolution reconstruction has a wide range of applications in many fields, such as video surveillance, remote sensing images, medical images, and cultural relic restoration. There are many effective super-resolution reconstruction algorithms, such as regularization-based algorithms, but in the existing super-resolution reconstruction algorithms, the image upsampling process and the deblurring process are separated. If the high-resolution image obtained by the up-sampling in the super-resolution reconstruction is simply deblurred, the advantages of the sequence image in the deblurring process are not utilized. Therefore, how to organically combine the image upsampling process and the deblurring process is still a major research challenge, and the research results also have high application value.

SUMMARY OF THE INVENTION

The purpose of the present invention is to design a regularization-based alternate minimization super-resolution reconstruction method based on the shortcomings of the regularization-based super-resolution reconstruction algorithm.

The invention analyzes the deficiencies of the existing regularization-based super-resolution reconstruction algorithm, and aims at the unreasonable solution process and the limitation of magnification in the super-resolution reconstruction algorithm, and the invention proposes an alternate minimization super-resolution reconstruction algorithm. construct algorithm. By adding a regular term that constrains the blur kernel, the upsampling process and the deblurring process are combined to make the algorithm more effective. In view of the insufficiency of dealing with magnification, it is proposed to subsample the original low-resolution image sequence to achieve Rational magnification; Aiming at the limitation of the super-resolution reconstruction algorithm on color image processing, it is proposed to add a regular term containing color information to the alternate minimization super-resolution reconstruction algorithm to achieve effective processing of color images. Different from the artifacts that often appear in existing algorithms, it is more suitable for direct observation by users. Therefore, the present invention is a regularization-based alternating minimization high-resolution image reconstruction method, the method comprising:

Step 1: Input a low-resolution image sequence, and determine that the magnification S, S ² is less than or equal to the number of input images;

Step 2: If S is an integer, go to Step 4 directly; otherwise, according to the magnification S, perform sub-sampling on the original low-resolution image sequence to obtain a down-sampled image;

Step 3: perform image registration based on mutual information on the down-sampled image obtained in step 2; wherein mutual information calculation means that other frames except the first frame image and the first frame perform mutual information calculation;

计算公式为：Given the reference image f ₁ (x, y) and the image to be matched f ₂ (x ₁ , y ₁ ), the information entropy of f ₁ (x, y) is H(f ₁ ), f ₂ (x ₁ , y _{1 )} ), the information entropy is H(f ₂ ), and their joint entropy is H(f ₁ , f ₂ ), then the mutual information between them

The calculation formula is:

Step 4. Initialize the high-resolution image and blur kernel, and construct a matrix blur kernel coefficient matrix N;

Initialize the high-resolution image, initialize the blur kernel to 0, and reconstruct the general model according to the super-resolution. For the super-resolution reconstruction model with a magnification of S=p/q, where p and q are two relatively prime positive integers , the degenerate model is represented by discretization, and we get:

表示低分辨率图像序列的离散分量

与一个大小为E×E滤波器的“Valid”卷积，E的取值范围为[3,15]；设模糊核F_k模糊核大小为H×H，H的取值范围为[3,15]，由高分辨率图像R、模糊核F_k和下采样矩阵D^p来表示矩阵ξ；Among them, k represents the kth frame image, Represents the value of the i-th row and j-column elements in the matrix represented by this discretization, D ^P represents the downsampling matrix, F _k represents the blur kernel, and R represents the high-resolution image; get a matrix

Discrete components representing a sequence of low-resolution images

Convolve with a "Valid" filter whose size is E×E, the value range of E is [3, 15]; the size of the blur kernel F _k is set to H×H, and the value range of H is [3, 15]. 15], the matrix ξ is represented by the high-resolution image R, the blur kernel F _k and the down-sampling matrix D ^p ;

ξ=S ^p Ωf

其中S^p表示与放大倍数相关的系数,根据放大倍数确定；放大倍数为S＝p/q，p，q为两个互质的整数，

表示R与大小为pE-p+H-1的滤波器卷积，C_pE-p+1{F₁}表示F₁与大小为pE-p+1的滤波器卷积，Ω中的下采样矩阵D^p的大小为(pE-p+1)²×E²；in

Among them, S ^p represents the coefficient related to the magnification, which is determined according to the magnification; the magnification is S=p/q, p, q are two relatively prime integers,

represents R convolution with a filter of size pE-p+H-1, C _pE-p+1 {F ₁ } represents F ₁ convolved with a filter of size pE-p+1, downsampling in Ω The size of the matrix D ^p is (pE-p+1) ² ×E ² ;

It is determined that f is of full rank, so to solve ξ=0, that is, to find the null space of ξ, that is, Ω=0; but from the size of the matrix, the dimension of the null space of ξ is greater than the dimension of the solution of Ω=0:

nullity(ξ)≥M=n(qE ² )-(pE-p+H-1) ²

where nullity(ξ) represents the dimension of the ξ0 space, M represents the dimension of the Ω=0 solution, and n represents the number of pictures;

Let μ _kn represent a filter of size E×E, d represent rows, n represent columns, and η _dn be the row vector decomposition of μ _dn with size 1×E, then use η _dn to represent the null space B of ξ is:

use represents the result of μ _dn after upsampling with a multiple of p,

Γ(R)通常来讲是一个高通滤波器；The final N matrix is obtained here; after the matrix N is obtained, the regular term for the fuzzy kernel constraint can be obtained

Γ(R) is usually a high-pass filter;

Step 5. If the input image is a color image, add a regular term of color information to the iterative equation, otherwise do not add it;

Step 6. Alternately estimate the clear image image R and the blur kernel F, and after the iteration is completed, output a high-resolution image;

The iterative equation is:

Γ(R)为一个高通滤波器；D_k表示第k帧图像的下采样矩阵，F_k表示第k帧图像的模糊核，λ、ω、φ均表示正则化系数，J(R)为对保真项约束的正则项，Q(F)表示对模糊核约束的正则项，W(R)表示对彩色信息约束的正则项，H_k表示第k帧图像的配准矩阵，L_k表示实际低分辨率序列，为步骤1或步骤3得到的图像，

表示p阶范数；in

Γ(R) is a high-pass filter; Dk represents the downsampling matrix of the _kth frame image, _Fk represents the blur kernel of the kth frame image, λ, ω, φ all represent the regularization coefficient, J(R) is the pair of The regular term constrained by the fidelity term, Q(F) represents the regular term constrained by the fuzzy kernel, W(R) represents the regular term constrained by the color information, H _k represents the registration matrix of the k-th frame image, and L _k represents the actual The low-resolution sequence, the images obtained in step 1 or step 3,

represents the p-order norm;

如果输入的低分辨率序列为彩色图像，则

If the input low-resolution sequence is a grayscale image, then the regularization coefficient

If the input low-resolution sequence is a color image, then

According to the results above, in the above formula, only F and R are unknown quantities, then F and R can be solved alternately, that is, an initial value R can be substituted, and F can be solved. To solve, repeat the above steps until the iteration termination condition is reached.

Further, in step 2, S=p/q, where p and q are two relatively prime positive integers, then sub-sampling the original low-resolution image with a multiple of q is performed.

其中代表求取梯度操作，f_r,f_g,f_b分别为高分辨率清晰图像的R，G，B三个分量，x,y表示代表行列号像素点的位置。Further, in the step 6, the added color information regular item is

in Represents the operation of obtaining the gradient, f _r , f _g , and f _b are the three components of R, G, and B of the high-resolution clear image, respectively, and x and y represent the position of the pixel representing the row and column number.

According to the theoretical analysis of sub-sampling the original low-resolution image sequence, the invention realizes the magnification of the rational number level, and makes up for the deficiency that the existing algorithm can only realize the magnification of the integer level. At the same time, the limitations of the existing super-resolution reconstruction algorithms for color image processing are explained, and a regular term containing color information is added to the alternate minimization super-resolution reconstruction algorithm to realize the optimal processing of color images.

Description of drawings

Figure 1 is the overall flow chart of the regularization-based alternate minimization super-resolution reconstruction algorithm.

Figure 2, Figure 3 and Figure 4 are schematic diagrams of grayscale image super-resolution, in which Figure 1 is the original image, Figure 2 is the result of 2x magnification, and Figure 3 is the result of 1.5x magnification.

Figure 5, Figure 6, Figure 7 and Figure 8 are schematic diagrams of color images and actual super-resolution, of which Figures 5 and 6 are the original images, and Figures 7 and 8 are the results of 2x magnification.

Detailed ways

A regularization-based alternating minimization high-resolution image reconstruction method, comprising:

Step 1: Input a low-resolution image sequence, and determine that the magnification S, S ² is less than or equal to the number of input images;

Step 2: If S is an integer, go to Step 4 directly; otherwise, according to the magnification S, perform sub-sampling on the original low-resolution image sequence to obtain a down-sampled image; S=p/q, where p and q are two is a relatively prime positive integer, then the original low-resolution image is sub-sampled by a multiple of q.

Step 3: perform image registration based on mutual information on the down-sampled image obtained in step 2; wherein mutual information calculation means that other frames except the first frame image and the first frame perform mutual information calculation;

计算公式为：Given the reference image f ₁ (x, y) and the image to be matched f ₂ (x ₁ , y ₁ ), the information entropy of f ₁ (x, y) is H(f ₁ ), f ₂ (x ₁ , y _{1 )} ), the information entropy is H(f ₂ ), and their joint entropy is H(f ₁ , f ₂ ), then the mutual information between them

The calculation formula is:

Step 4. Initialize the high-resolution image and blur kernel, and construct a matrix blur kernel coefficient matrix N;

Initialize the high-resolution image, initialize the blur kernel to 0, and reconstruct the general model according to the super-resolution. For the super-resolution reconstruction model with a magnification of S=p/q, where p and q are two relatively prime positive integers , the degenerate model is represented by discretization, and we get:

表示这个离散化表示的矩阵中第i行j列元素的值，D^P表示下采样矩阵，F_k表示模糊核，R表示高分辨率图像；得到一个矩阵

表示低分辨率图像序列的离散分量

与一个大小为E×E滤波器的“Valid”卷积，E的取值范围为[3,15]；设模糊核F_k模糊核大小为H×H，H的取值范围为[3,15]，由高分辨率图像R、模糊核F_k和下采样矩阵D^p来表示矩阵ξ；Among them, k represents the kth frame image,

Represents the value of the i-th row and j-column elements in the matrix represented by this discretization, D ^P represents the downsampling matrix, F _k represents the blur kernel, and R represents the high-resolution image; get a matrix

Discrete components representing a sequence of low-resolution images

Convolve with a "Valid" filter whose size is E×E, the value range of E is [3, 15]; the size of the blur kernel F _k is set to H×H, and the value range of H is [3, 15]. 15], the matrix ξ is represented by the high-resolution image R, the blur kernel F _k and the down-sampling matrix D ^p ;

ξ=S ^p Ωf

其中S^p表示与放大倍数相关的系数,根据放大倍数确定；放大倍数为S＝p/q，p，q为两个互质的整数，

表示R与大小为pE-p+H-1的滤波器卷积，C_pE-p+1{F₁}表示F₁与大小为pE-p+1的滤波器卷积，Ω中的下采样矩阵D^p的大小为(pE-p+1)²×E²；in

Among them, S ^p represents the coefficient related to the magnification, which is determined according to the magnification; the magnification is S=p/q, p, q are two relatively prime integers,

represents R convolution with a filter of size pE-p+H-1, C _pE-p+1 {F ₁ } represents F ₁ convolved with a filter of size pE-p+1, downsampling in Ω The size of the matrix D ^p is (pE-p+1) ² ×E ² ;

It is determined that f is of full rank, so to solve ξ=0, that is, to find the null space of ξ, that is, Ω=0; but from the size of the matrix, the dimension of the null space of ξ is greater than the dimension of the solution of Ω=0:

nullity(ξ)≥M=n(qE ² )-(pE-p+H-1) ²

where nullity(ξ) represents the dimension of the ξ0 space, M represents the dimension of the Ω=0 solution, and n represents the number of pictures;

Let μ _kn represent a filter of size E×E, d represent rows, n represent columns, and η _dn be the row vector decomposition of μ _dn with size 1×E, then use η _dn to represent the null space B of ξ is:

表示μ_dn在倍数为p的上采样后的结果，use

represents the result of μ _dn after upsampling with a multiple of p,

Γ(R)通常来讲是一个高通滤波器；The final N matrix is obtained here; after the matrix N is obtained, the regular term for the fuzzy kernel constraint can be obtained

Γ(R) is usually a high-pass filter;

Step 5. If the input image is a color image, add a regular term of color information to the iterative equation, otherwise do not add it;

Step 6. Alternately estimate the clear image image R and the blur kernel F, and after the iteration is completed, output a high-resolution image;

The iterative equation is:

Γ(R)为一个高通滤波器；D_k表示第k帧图像的下采样矩阵，F_k表示第k帧图像的模糊核，λ、ω、φ均表示正则化系数，J(R)为对保真项约束的正则项，Q(F)表示对模糊核约束的正则项，W(R)表示对彩色信息约束的正则项，H_k表示第k帧图像的配准矩阵，L_k表示实际低分辨率序列，为步骤1或步骤3得到的图像，

表示p阶范数；in

Γ(R) is a high-pass filter; Dk represents the downsampling matrix of the _kth frame image, _Fk represents the blur kernel of the kth frame image, λ, ω, φ all represent the regularization coefficient, J(R) is the pair of The regular term constrained by the fidelity term, Q(F) represents the regular term constrained by the fuzzy kernel, W(R) represents the regular term constrained by the color information, H _k represents the registration matrix of the k-th frame image, and L _k represents the actual The low-resolution sequence, the images obtained in step 1 or step 3,

represents the p-order norm;

If the input low-resolution sequence is a grayscale image, then the regularization coefficient If the input low-resolution sequence is a color image, then

其中

代表求取梯度操作，f_r,f_g,f_b分别为高分辨率清晰图像的R，G，B三个分量，x,y表示代表行列号像素点的位置；The added color information regular item is

in

Represents the gradient operation, f _r , f _g , and f _b are the three components of R, G, and B of the high-resolution clear image, respectively, and x, y represent the position of the pixel representing the row and column number;

According to the results above, in the above formula, only F and R are unknown quantities, then F and R can be solved alternately, that is, an initial value R can be substituted, and F can be solved. To solve, repeat the above steps until the iteration termination condition is reached.

Claims (3)

1. A regularization-based alternating minimization high-resolution image reconstruction method, the method comprising: Step 1: Input a low-resolution image sequence, and determine that the magnification S, S ² is less than or equal to the number of input images; Step 2: If S is an integer, go to Step 4 directly; otherwise, according to the magnification S, perform sub-sampling on the original low-resolution image sequence to obtain a down-sampled image; Step 3: perform image registration based on mutual information on the down-sampled image obtained in step 2; wherein mutual information calculation means that other frames except the first frame image and the first frame perform mutual information calculation; Given the reference image f ₁ (x, y) and the image to be matched f ₂ (x ₁ , y ₁ ), the information entropy of f ₁ (x, y) is H(f ₁ ), f ₂ (x ₁ , y _{1 )} ), the information entropy is H(f ₂ ), and their joint entropy is H(f ₁ , f ₂ ), then the mutual information between them

The calculation formula is:

Step 4. Initialize the high-resolution image and blur kernel, and construct a matrix blur kernel coefficient matrix N; Initialize the high-resolution image, initialize the blur kernel to 0, and reconstruct the general model according to the super-resolution. For the super-resolution reconstruction model with a magnification of S=p/q, where p and q are two relatively prime positive integers , the degenerate model is represented by discretization, and we get:

Among them, k represents the kth frame image,

Represents the value of the i-th row and j-column elements in the matrix represented by this discretization, D ^P represents the downsampling matrix, F _k represents the blur kernel, and R represents the high-resolution image; get a matrix

Discrete components representing a sequence of low-resolution images

Convolve with a "Valid" filter whose size is E×E, the value range of E is [3, 15]; the size of the blur kernel F _k is set to H×H, and the value range of H is [3, 15]. 15], the matrix ξ is represented by the high-resolution image R, the blur kernel F _k and the down-sampling matrix D ^p ; ξ=S ^p Ωf in

Among them, S ^p represents the coefficient related to the magnification, which is determined according to the magnification; the magnification is S=p/q, p, q are two relatively prime integers,

represents R convolution with a filter of size pE-p+H-1, C _pE-p+1 {F ₁ } represents F ₁ convolved with a filter of size pE-p+1, downsampling in Ω The size of the matrix D ^p is (pE-p+1) ² ×E ² ; It is determined that f is of full rank, so to solve ξ=0, that is, to find the null space of ξ, that is, Ω=0; but from the size of the matrix, the dimension of the null space of ξ is greater than the dimension of the solution of Ω=0: nullity(ξ)≥M=n(qE ² )-(pE-p+H-1) ² where nullity(ξ) represents the dimension of the ξ0 space, M represents the dimension of the Ω=0 solution, and n represents the number of pictures; Let μ _kn represent a filter of size E×E, d represent rows, n represent columns, and η _dn be the row vector decomposition of μ _dn with size 1×E, then use η _dn to represent the null space B of ξ is: use

represents the result of μ _dn after upsampling with a multiple of p,

The final N matrix is obtained here; after the matrix N is obtained, the regular term for the fuzzy kernel constraint can be obtained

Γ(R) is usually a high-pass filter; Step 5. If the input image is a color image, add a regular term of color information to the iterative equation, otherwise do not add it; Step 6. Alternately estimate the clear image image R and the blur kernel F, and after the iteration is completed, output a high-resolution image; The iterative equation is:

in

Γ(R) is a high-pass filter; Dk represents the downsampling matrix of the _kth frame image, _Fk represents the blur kernel of the kth frame image, λ, ω, φ all represent the regularization coefficient, J(R) is the pair of The regular term constrained by the fidelity term, Q(F) represents the regular term constrained by the fuzzy kernel, W(R) represents the regular term constrained by the color information, H _k represents the registration matrix of the k-th frame image, and L _k represents the actual The low-resolution sequence, the images obtained in step 1 or step 3,

represents the p-order norm; If the input low-resolution sequence is a grayscale image, then the regularization coefficient If the input low-resolution sequence is a color image, then

According to the results above, in the above formula, only F and R are unknown quantities, then F and R can be solved alternately, that is, an initial value R can be substituted, and F can be solved. To solve, repeat the above steps until the iteration termination condition is reached. 2. a kind of regularization-based alternating minimization high-resolution image reconstruction method as claimed in claim 1, is characterized in that in described step 2, S=p/q, wherein p and q are two relative primes If it is a positive integer, then subsample the original low-resolution image with a multiple of q. 3. a kind of regularization-based alternate minimization high-resolution image reconstruction method as claimed in claim 1, is characterized in that in described step 6, the color information regular term that adds is

in

Represents the gradient operation, f _r , f _g , and f _b are the R, G, and B components of the high-resolution clear image, respectively, and x, y represent the position of the pixel representing the row and column number.

Images