CN103208109A - Local restriction iteration neighborhood embedding-based face hallucination method - Google Patents

Abstract

A face illusion method based on local constraint iterative neighborhood embedding, which establishes high and low resolution image block sets as high and low resolution image block dictionaries; Estimate the high-resolution image block, and find the nearest K image blocks in the high-resolution image block dictionary of the corresponding position, and use the corresponding K low-resolution image blocks to linearly represent the input low-resolution image block to obtain the weight coefficient; use The weight coefficient reconstructs K adjacent high-resolution image blocks into a new estimated high-resolution image block, and repeats until the most satisfactory estimated high-resolution image block is obtained; according to the positional relationship of the low-resolution image blocks, it is fused into High resolution images. The present invention considers two manifold structures at the same time on the basis of position prior and local manifold constraints, and uses iteration to continuously update the K nearest neighbors and reconstruction weights on the result of the last reconstruction, and obtains higher quality, closer to real The situation is closer to the reconstruction effect.

Description

A face illusion method based on iterative neighborhood embedding with local constraints

technical field

The invention relates to the field of image super-resolution, in particular to a face illusion method based on local constraint iterative neighborhood embedding.

Background technique

In the past 20 years, face recognition technology has developed rapidly. At the same time, due to the limited network bandwidth and server storage of the video surveillance system, the resolution of the captured facial images is low, making the face information that can be provided very limited, which has become one of the most challenging problems in biometric technology. Recently, super-resolution technology has been used to process low-resolution (Low-Resolution, LR) images, which can generate a sequence of low-resolution images or a single frame of low-resolution images that can provide more faces for the subsequent recognition process. High-Resolution (HR) images of details.

In 2000, Baker and Kanade proposed a face hallucination in Document 1 (S.Baker and T.Kanade. Hallucinating faces. In FG, Grenoble, France, Mar. 2000, 83-88.) The method uses the prior information of the face images in the training set to obtain the high-resolution face images corresponding to the low-resolution face images through the learning method. This is a pioneering work in the field of face illusion technology. After that, many different methods and models were introduced, among which the most representative technology is Chang et al. in literature 2 (H. Chang, D. Yeung, and Y. Xiong. Super-resolution through neighbor embedding[A]. In Proc.IEEE CVPR'04[C].Washington,2004.275–282.) A method based on manifold learning, they assume that high-resolution image blocks and low-resolution image blocks have the same local geometric structure, Then, locally linear embedding (LLE) is used to learn the correspondence between high and low resolution image patch manifolds. Theoretical research on face analysis and synthesis shows that the location information of face images is very important in face analysis and synthesis. Inspired by this, Ma et al. Hallucinating face by position-patch. Pattern Recognition, 43(6):3178–3194, 2010.) and literature 4 (X.Ma, J. Zhang, and C. Qi, “Position-based face hallucination method,” in Proc. IEEE Conf.on Multimedia and Expo (ICME), 2009, pp.290–293.) proposes a face illusion method based on location blocks, for a given low-resolution face image at a certain position through the image block All image blocks at the same position in the training set face images are linearly synthesized. However, since this method uses the least square method to solve, when the number of images in the training sample is larger than the dimension of the image block, the representation coefficient of the image block is not unique. For this reason, in 2011, Jung et al. published in literature 5 (C.Jung, L.Jiao, B.Liu, and M.Gong, "Position-Patch Based Face Hallucination Using Convex Optimization," IEEE Signal Process. Lett., vol. 18, no.6, pp.367–370, 2011.) using sparse regularization method to obtain the optimal reconstruction weights for face phantom. Recently, in Patent 1 (Hu Ruimin, Jiang Junjun, Wang Bing, Han Zhen, Huang Kebin, Lu Tao, Wang Yimin, a face super-resolution reconstruction method based on local constraint representation, patent application number: 201110421452.3) further improved the position-based block The face phantom method, in the process of reconstructing the image block, replaces the sparse constraint in literature 5 with the local geometric constraint of the manifold, so that the reconstruction result has both sparsity and locality. This method of locally constrained representation face hallucination is by far the best performing method.

Whether using sparse representation or local constraints, the methods mentioned above perform face hallucination by exploring reasonable prior knowledge, finding the most representative image patches and obtaining optimal weights. Therefore, how to find reasonable K nearest neighbor sample image blocks and obtain optimal weights are the two most critical links in face illusion technology. All the methods mentioned above only consider the manifold of low-resolution image blocks, but ignore the geometric structure information of high-resolution image blocks, making the reconstruction results lack of reliability and discrimination.

Contents of the invention

The purpose of the present invention is to provide a face illusion method based on local constraint iterative neighborhood embedding. The entire training sample image block is used as a dictionary, and the local geometric structure characteristics of the low-resolution manifold structure and the high-resolution manifold structure are considered, which makes up for the lack of only considering a certain manifold structure in the past. At the same time, after multiple iterations steps to make the reconstruction effect closer to the original high-resolution image.

In order to achieve the above object, the technical solution adopted in the present invention is a method of face illusion based on local constraint iterative neighborhood embedding, comprising the following steps:

Step 1. Up-sampling the input low-resolution face image to obtain the estimated high-resolution face image. For the input low-resolution face image, the estimated high-resolution face image, and the low-resolution training set All low-resolution face sample images and all high-resolution face sample images in the high-resolution training set are divided into overlapping image blocks;

Step 2, for each image block in the input low-resolution face image, perform the following steps to obtain a reconstructed high-resolution image block,

Step 2.1, take the image block at the corresponding position of each low-resolution face sample image in the low-resolution training set as a sample point, and establish a low-resolution face sample block space; take each high-resolution face sample image in the high-resolution training set The image block at the corresponding position of the sample image is used as a sample point to establish a high-resolution face sample block space; the image block at the corresponding position of the estimated high-resolution face image is taken to obtain an estimated high-resolution image block;

Step 2.2, calculate the K nearest image blocks of the current reconstructed high-resolution image block in the high-resolution face sample block space obtained in the previous iteration, and find the K image blocks in the low-resolution face sample block space K image blocks in , when step 2.2 is executed for the first time, the estimated high-resolution image block obtained in step 2.1 is used as the current reconstructed high-resolution image block; and K image blocks in the low-resolution face sample block space are used Linearly reconstruct the input low-resolution image block to obtain the optimal weight coefficient; use the optimal weight coefficient and K image blocks in the high-resolution face sample block space to linearly reconstruct to obtain the Reconstructing high-resolution image blocks; K is a preset value;

Step 2.3, according to the current reconstructed high-resolution image block obtained in step 2.2, return to repeat step 2.2 until the number of iterations reaches the preset value;

Step 3, superimpose the corresponding reconstructed high-resolution image blocks of all image blocks in the input low-resolution face image according to the position, and then divide by the number of overlaps of each pixel position to reconstruct a high-resolution face image.

预估高分辨率人脸图像Y^t(0)划分所得图像块集为

对高分辨率训练集中所有高分辨率人脸样本图像分别划分得到高分辨率图像块集Y＝{y_ij|1≤i≤N,1≤j≤M}，对低分辨率训练集中所有低分辨率人脸样本图像分别划分得到低分辨率图像块集X＝{x_ij|1≤i≤N,1≤j≤M}；其中，标识i表示高分辨率训练集中高分辨率人脸样本图像的序号和低分辨率训练集中低分辨率人脸样本图像的序号，标识j表示图像上块位置序号；

是预估高分辨率人脸图像Y^t(1)位置j处的图像块，

是输入的低分辨率人脸图像X^t位置j处的图像块；y_ij是高分辨率训练集中第i张图像位置j处的图像块，x_ij是低分辨率训练集中第i张图像位置j处的图像块；低分辨率训练集中低分辨率人脸样本图像的个数和高分辨率训练集中高分辨率人脸样本图像的个数都记为N，M为每幅图像划分图像块的块数；Moreover, it is assumed that the image block set obtained by dividing the input low-resolution face image X ^t is

It is estimated that the image block set obtained by dividing the high-resolution face image Y ^t (0) is

Divide all high-resolution face sample images in the high-resolution training set to obtain a high-resolution image block set Y={y _ij |1≤i≤N, 1≤j≤M}, and for all low-resolution training sets The high-resolution face sample images are respectively divided to obtain a low-resolution image block set X={x _ij |1≤i≤N, 1≤j≤M}; where, the identifier i represents the high-resolution face sample in the high-resolution training set The serial number of the image and the serial number of the low-resolution face sample image in the low-resolution training set, and the mark j represents the serial number of the block position on the image;

is the image block at position j of the estimated high-resolution face image Y ^t (1),

is the image block at position j of the input low-resolution face image X ^t ; y _ij is the image block at position j of the i-th image in the high-resolution training set, x _ij is the position of the i-th image in the low-resolution training set The image block at j; the number of low-resolution face sample images in the low-resolution training set and the number of high-resolution face sample images in the high-resolution training set are recorded as N, and M is divided into image blocks for each image the number of blocks;

执行以下子步骤，In step 2, any image block in the input low-resolution face image

Perform the following sub-steps,

建立低分辨率人脸样本块空间X_j＝{x_ij|1≤i≤N}作为低分辨率图像块字典，建立高分辨率人脸样本块空间Y_j＝{y_ij|1≤i≤N}作为高分辨率图像块字典，寻找相应位置预估高分辨率图像块

其中 ${\{y}_j^t\left(1\right)=\mathrm{Bicubic}\left(x_j^t\right)|1\≤j\≤M\};$ Step 2.1, let the initial value of the number of iterations p be 1; for an image block at any position in the input low-resolution face image

Establish the low-resolution face sample block space X _j ={x _ij |1≤i≤N} as the low-resolution image block dictionary, and establish the high-resolution face sample block space Y _j ={y _ij |1≤i≤ N} as a high-resolution image block dictionary, looking for the corresponding position to estimate the high-resolution image block

in ${\{\mathrm{the\; y}}_j^t\left(1\right)=\mathrm{Bicubic}\left(x_j^t\right)|1\≤j\≤m\};$

Step 2.2, using the reconstructed high-resolution image block obtained by performing step 2.2 in the previous iteration, to calculate the reconstructed high-resolution image block of this iteration, including the following sub-steps:

计算与相应位置的高分辨率图像块字典Y_j＝{y_ij|1≤i≤N}中各图像块的距离，并寻找距离最小的K个图像块如下，Step 2.2.1, according to the current reconstructed high-resolution image patch

Calculate the distance from each image block in the high-resolution image block dictionary Y _j ={y _ij |1≤i≤N} at the corresponding position, and find the K image blocks with the smallest distance as follows,

$\mathrm{<span\; class="notranslate">\; dist</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; N</span>}$

${\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; support</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; dist</span>}<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; )</span>$

与高分辨率图像块字典中各图像块y_ij的距离，R^N表示N维实数空间；dist|_K表示dist(p)中最小的K个值，

是当前的高分辨率图像块与高分辨率图像块字典中距离最小的K个图像块的索引集合；Among them, dist(p)∈R ^N , dist(p) represents the reconstructed high-resolution image block

The distance from each image block y _ij in the high-resolution image block dictionary, R ^N represents the N-dimensional real number space; dist| _K represents the smallest K values in dist(p),

is the current high-resolution image patch An index set of K image blocks with the smallest distance in the high-resolution image block dictionary;

采用预估高分辨率图像块

之后执行步骤2.2.1时，当前的重构高分辨率图像块

为上一次迭代时执行步骤2.2.3得到的重构高分辨率图像块；When step 2.2.1 is performed for the first time, the current reconstructed high-resolution image patch

Using estimated high-resolution image patches

After performing step 2.2.1, the current reconstructed high-resolution image block

The reconstructed high-resolution image block obtained by performing step 2.2.3 during the previous iteration;

中K个图像块y_k分别在低分辨率图像块字典中的对应图像块x_k，并对图像块

进行线性重构，得到最优权值系数

如下式，Step 2.2.2, find the set obtained in step 2.2.1

The corresponding image blocks x _k of the K image blocks y _k in the low-resolution image block dictionary, and the image blocks

Perform linear reconstruction to obtain the optimal weight coefficient

as follows,

返回关于权值系数w(p)的函数在得到最小值时w(p)的取值，w_k(p)表示权值系数w(p)中对应x_k的分量，τ是平衡重建误差和局部约束的正则化参数；in,

Returns the value of w(p) when the function about the weight coefficient w(p) obtains the minimum value, w _k (p) represents the component corresponding to x _k in the weight coefficient w(p), τ is the balance reconstruction error and Locally constrained regularization parameters;

In step 2.2.3, the reconstructed high-resolution image block of this iteration is reconstructed by the following formula,

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; w</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}$

是步骤2.2.2所得最优权值系数

中对应图像块y_k的分量；in,

is the optimal weight coefficient obtained in step 2.2.2

The component corresponding to the image block y _k in ;

Step 2.3, judge whether the number of iterations reaches the preset value, otherwise set p=p+1, return to repeat step 2.2 for the next iteration according to the reconstructed high-resolution image block obtained in step 2.2.3 in this iteration; In this iteration, the reconstructed high-resolution image block obtained in step 2.2.3 is used as the final reconstructed high-resolution image block, and the iteration ends.

A face phantom method based on local constraint iterative neighborhood embedding proposed by the present invention is different from previous methods that only consider the manifold structure of low-resolution images, and increases the impact of high-resolution image manifold structure on reconstruction coefficients. Constraints, while revealing the inherent structural similarity of the high- and low-resolution face manifold spaces, it also makes use of the essential differences between the two manifold spaces to make the reconstruction results more discriminative; the reconstruction weight and The neighbor points are updated, and the neighbor points and reconstruction coefficients that are more representative of the target high-resolution image block are selected, and finally a high-resolution image that is closer to the real situation is obtained. This method improves the traditional neighborhood embedding method, makes the representation coefficients of the input image blocks more accurate, and finally obtains higher-quality high-resolution face images.

Description of drawings

Fig. 1 is a flowchart of an embodiment of the present invention.

Detailed ways

The technical scheme of the present invention can adopt software technology to realize automatic flow operation. The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. Referring to Fig. 1, the specific steps of the embodiment of the present invention are as follows:

Step 1, initialize the input low-resolution face image, that is, Bicubic upsampling to obtain the estimated high-resolution face image, and input low-resolution face image, estimated high-resolution face image, low-resolution All low-resolution face sample images in the high-resolution training set and all high-resolution face sample images in the high-resolution training set are divided into overlapping image blocks in the same way;

The low-resolution training set and the high-resolution training set provide preset training sample pairs, the low-resolution training set contains low-resolution face sample images, and the high-resolution training set contains high-resolution face sample images. In the embodiment, all high-resolution human face sample images are aligned and registered human face images (eyes or mouth parts can be manually aligned in advance), and the pixel size is 112×100. Each low-resolution face sample image in the low-resolution training set is obtained from a high-resolution face sample image in the high-resolution training set with 4×4 smooth filtering and 4 times downsampling, and the pixel size of the low-resolution image is 28 ×25. Correspondingly, the pixel size of the input low-resolution face image is 28×25, and it can also be pre-aligned and registered with the face position of the high-resolution face sample image. The estimated pixel size of the high-resolution face image is 112× 100. Overlap division is a common technique in the art, and those skilled in the art can specify the overlapping pixel size during specific implementation. In the embodiment, the high-resolution image block size of all high-resolution face sample images and estimated high-resolution face images is set to 12×12, and the overlapping pixel value is set to 4. All low-resolution face sample images and The low-resolution image patch size of the input low-resolution face image is 3×3, and the overlapping pixel value is 1.

M为图像块数。对输入低分辨率人脸图像进行Bicubic上采样，得到预估高分辨率人脸图像Y^t(1)＝Bicubic(X^t)，对其用相同的方法划分相互重叠的图像块后得到与低分辨率图像块集对应的预估高分辨率人脸图像块集

对高、低分辨率训练集中人脸样本图像用相同的方法分别划分相互重叠的图像块后得到高分辨率图像块集Y＝{y_ij|1≤i≤N,1≤j≤M}和低分辨率图像块集X＝{x_ij|1≤i≤N,1≤j≤M}，其中，标识i表示高分辨率训练集中高分辨率人脸样本图像的序号和低分辨率训练集中低分辨率人脸样本图像的序号，标识j表示每张图像上的块位置序号。是预估高分辨率人脸图像Y^t(1)位置j处的图像块，

是输入的低分辨率人脸图像X^t位置j处的图像块，y_ij、x_ij分别是高、低分辨率训练集中第i张图像位置j处的图像块，低分辨率训练集中低分辨率人脸样本图像的个数和高分辨率训练集中高分辨率人脸样本图像的个数都记为N，M为每幅图像划分图像块的块数；In an embodiment, the image block set formed after dividing the input low-resolution face image X ^t into overlapping image blocks is

M is the number of image blocks. Bicubic upsampling is performed on the input low-resolution face image to obtain an estimated high-resolution face image Y ^t (1)=Bicubic(X ^t ), and the same method is used to divide the overlapping image blocks to obtain the low The estimated high-resolution face image patch set corresponding to the high-resolution image patch set

For the face sample images in the high-resolution and low-resolution training sets, use the same method to divide overlapping image blocks to obtain the high-resolution image block set Y={y _ij |1≤i≤N, 1≤j≤M} and Low-resolution image block set X={x _ij |1≤i≤N, 1≤j≤M}, wherein, the identifier i represents the serial number of the high-resolution face sample image in the high-resolution training set and the sequence number of the high-resolution face sample image in the low-resolution training set The serial number of the low-resolution face sample image, and the identifier j indicates the serial number of the block position on each image. is the image block at position j of the estimated high-resolution face image Y ^t (1),

is the image block at position j of the input low-resolution face image X ^t , y _ij and x _ij are the image block at position j of the i-th image in the high- and low-resolution training set, respectively, and the low-resolution training set in the low-resolution training set The number of high-resolution human face sample images and the number of high-resolution human face sample images in the high-resolution training set are all recorded as N, and M is the number of blocks divided into image blocks for each image;

执行以下子步骤得到相应的重构高分辨率图像块：Step 2, for any image block in the input low-resolution face image

Perform the following sub-steps to obtain the corresponding reconstructed high-resolution image patches:

Step 2.1, take the image block corresponding to the position of each low-resolution face sample image in the low-resolution training set as a sample point, establish a low-resolution face sample block space, and take each high-resolution face sample image in the high-resolution training set The image block at the corresponding position of the sample image is used as the sample point, and the high-resolution face sample block space is established, and the image block at the corresponding position of the estimated high-resolution face image is taken to obtain the estimated high-resolution image block; let the number of iterations p The initial value of is 1;

其中 ${\{y}_j^t\left(1\right)=\mathrm{Bicubic}\left(x_j^t\right)|1\&le;j\&le;M\};$ In an embodiment, for a certain position image block in the input low-resolution face image Establish the low-resolution face sample block space X _j ={x _ij |1≤i≤N} as the low-resolution image block dictionary, and the high-resolution face sample block space Y _j ={y _ij |1≤i≤N } as a high-resolution image block dictionary, looking for the corresponding position to estimate the high-resolution image block

in ${\{\mathrm{the\; y}}_j^t\left(1\right)=\mathrm{Bicubic}\left(x_j^t\right)|1\&le;j\&le;m\};$

Step 2.2, use the reconstructed high-resolution image block obtained in step 2.2 in the previous iteration to calculate its corresponding target for this iteration, that is, the reconstructed high-resolution image block for this iteration, including the following sub-steps:

Step 2.2.1, according to the current reconstructed high-resolution image patch Calculate the distance between it and each image block in the high-resolution image block dictionary Y _j ={y _ij |1≤i≤N} at the corresponding position, and find the K image blocks with the smallest distance, that is, the K nearest neighbors, the method as follows,

$\mathrm{<span\; class="notranslate">\; dist</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; N</span>}$

${\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; support</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; dist</span>}<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; )</span>$

是当前的重构高分辨率图像块

与高分辨率图像块字典中距离最小的K个图像块的索引集合，p是当前的迭代次数。K可由本领域技术人员预设数值，本实施例中，K设为150。Among them, dist(p)∈R ^N , dist(p) represents the reconstructed high-resolution image block The distance from each image block y _ij in the high-resolution image block dictionary, R ^N represents the N-dimensional real number space; dist| _K represents the smallest K values in dist(p), ||·|| ₂ represents the two-norm,

is the current reconstructed high-resolution image patch

Index set of K image blocks with the smallest distance in the high-resolution image block dictionary, p is the current iteration number. K can be preset by those skilled in the art. In this embodiment, K is set to 150.

采用对输入图像的图像块

获得的预估高分辨率人脸图像Y^t(0)中相应预估高分辨率图像块

之后执行步骤2.2.1时，当前的重构高分辨率图像块

即上一次迭代时执行步骤2.2.3得到的重构高分辨率图像块。When step 2.2.1 is performed for the first time, the current reconstructed high-resolution image patch

Image blocks of the input image are used

The corresponding estimated high-resolution image block in the obtained estimated high-resolution face image Y ^t (0)

After performing step 2.2.1, the current reconstructed high-resolution image block

That is, the reconstructed high-resolution image block obtained by performing step 2.2.3 in the previous iteration.

中K个高分辨率的图像块分别在低分辨率图像块字典中对应的低分辨率的图像块，并利用它们对图像块

进行线性重构，得到重构的最优权值系数

方法如下：Step 2.2.2, find the set obtained in step 2.2.1

The K high-resolution image blocks correspond to the low-resolution image blocks in the low-resolution image block dictionary, and use them to map the image block

Perform linear reconstruction to obtain the optimal weight coefficient for reconstruction

Methods as below:

x_k是K个高分辨率图像块之一y_k所对应的低分辨率的图像块，w_k(p)表示权值系数w(p)中对应x_k的分量，“ο”表示两个向量之间的内积运算，

表示对二范数||·||₂的结果求平方，τ是平衡重建误差和局部约束的正则化参数，τ建议取值1e-5。in, Returns the value of w(p) when the function about the weight coefficient w(p) gets the minimum value, that is, the required optimal weight coefficient

x _k is the low-resolution image block corresponding to one of the K high-resolution image blocks y _k , w _k (p) represents the component corresponding to x _k in the weight coefficient w (p), and "ο" represents two Inner product operation between vectors,

Represents the square of the result of the two-norm ||·|| ₂ , τ is a regularization parameter to balance the reconstruction error and local constraints, and τ is recommended to take a value of 1e-5.

后，可以通过下式来重建本次迭代的重构高分辨率图像块，即新的当前重构高分辨率图像块，Step 2.2.3, after obtaining the optimal weight coefficient

Finally, the reconstructed high-resolution image block of this iteration can be reconstructed by the following formula, that is, the new current reconstructed high-resolution image block,

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; w</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}$

是本次迭代中执行步骤2.2.2所求的最优权值系数

中对应某一个高分辨率的图像块y_k的分量。in,

is the optimal weight coefficient obtained by performing step 2.2.2 in this iteration

Corresponding to the component of a certain high-resolution image block y _k .

Step 2.3, judge whether the number of iterations reaches the preset value, otherwise set p=p+1, return to repeat step 2.2 for the next iteration according to the reconstructed high-resolution image block obtained in step 2.2.3 in this iteration; In this iteration, the reconstructed high-resolution image block obtained in step 2.2.3 is used as the final reconstructed high-resolution image block, and the iteration ends;

In step 3, all weighted reconstructed high-resolution image blocks are superimposed according to position, and then divided by the number of overlaps of each pixel position to reconstruct a high-resolution face image.

The embodiment of the present invention mainly involves three parameters, that is, the number of nearest projection points K, the regularization parameter τ, and the number of iterations. Experiments show that when K is 150, better reconstruction results can be obtained; when the regularization parameter τ is between 1e-6 and 1e-3, our method can obtain stable performance. At the same time, in order to ensure the reconstruction error As small as possible, the parameter τ is set to 1e-5 to obtain the best effect; with the increase of the number of iterations, the PSNR value obtained from the experiment is also increasing. When the number of iterations increases to a certain extent, the PSNR value tends to Nearly stable, in order to achieve the best results as possible while reducing computational complexity, it is recommended to set the number of iterations to 6.

In order to verify the effectiveness of the present invention, the CAS-PEAL-R1 large-scale Chinese face database (document 6: W.Gao, B.Cao, S.Shan, X.Chen, et al. The CAS-PEAL Large-Scale Chinese Face Database and BaselineEvaluations[J].IEEE Trans.SMC(Part A),2008,38(1):149-161) for experiments, using all 1040 individuals with neutral expressions and frontal face images under normal lighting . Cut out the face area and crop it into 112×100 pixels, then manually mark five feature points on the face (the center of the eyes, the tip of the nose, and the two corners of the mouth) and perform affine transformation alignment to obtain the original high-resolution face image. The low-resolution face image is obtained by downsampling the high-resolution face image by 4 times Bicubic and then upsampling it by 4 times Bicubic. 1000 images are randomly selected as training samples, and the remaining 40 images are used as test images. We compared the reconstruction effect obtained by the present invention with Wang's global face method (document 7, X.Wang and X.Tang, "Hallucinating face by eigen transformation," IEEE Trans.Systems, Man, and Cybernetics.Part C, vol.35, no.3, pp.425–434, 2005.) and some methods based on block locations, such as neighborhood embedding method (NE, literature 2), least squares method (LSR, literature 3), sparse representation ( SR, Document 5) and Local Constraint Representation (LcR, Patent 1), etc.

The experiment uses Peak Signal to Noise Ratio (PSNR) to measure the pros and cons of the comparison algorithm, and SSIM is an index to measure the similarity between two images. The closer the value is to 1, the better the image reconstruction effect is. Comparing the average PSNR and SSIM values obtained by processing all 40 test images by the above method, the average PSNR values of global face, neighborhood embedding, least squares, sparse representation, local constraint representation and the method of the present invention are 26.53, 27.90 , 28.17, 28.27, 28.84, 29.33; the average SSIM values of methods such as global face, neighborhood embedding, least squares, sparse representation, local constraints and the method of the present invention are 0.8247, 0.8868, 0.8975, 0.8968, 0.9083, 0.9140. Compared with the best algorithm (patent 1) in the comparative method, the method of the present invention improves PSNR and SSIM values by 0.49 dB and 0.006 respectively. It can be seen that, compared with other existing methods, the effect of the method of the present invention has been significantly improved.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which the present invention belongs can make various modifications or supplements to the described specific embodiments or adopt similar methods to replace them, but they will not deviate from the spirit of the present invention or go beyond the definition of the appended claims range.

Claims (2)

1. a kind of face illusion construction method based on local constraint iterative neighborhood embedding, it is characterized in that, comprises the steps: Step 1. Up-sampling the input low-resolution face image to obtain the estimated high-resolution face image. For the input low-resolution face image, the estimated high-resolution face image, and the low-resolution training set All low-resolution face sample images and all high-resolution face sample images in the high-resolution training set are divided into overlapping image blocks; Step 2, for each image block in the input low-resolution face image, perform the following steps to obtain a reconstructed high-resolution image block, Step 2.1, take the image block at the corresponding position of each low-resolution face sample image in the low-resolution training set as a sample point, and establish a low-resolution face sample block space; take each high-resolution face sample image in the high-resolution training set The image block at the corresponding position of the sample image is used as a sample point to establish a high-resolution face sample block space; the image block at the corresponding position of the estimated high-resolution face image is taken to obtain an estimated high-resolution image block; Step 2.2, calculate the K nearest image blocks of the current reconstructed high-resolution image block in the high-resolution face sample block space obtained in the previous iteration, and find the K image blocks in the low-resolution face sample block space K image blocks in , when step 2.2 is executed for the first time, the estimated high-resolution image block obtained in step 2.1 is used as the current reconstructed high-resolution image block; and K image blocks in the low-resolution face sample block space are used Linearly reconstruct the input low-resolution image block to obtain the optimal weight coefficient; use the optimal weight coefficient and K image blocks in the high-resolution face sample block space to linearly reconstruct to obtain the optimal weight coefficient of this iteration. Reconstructing high-resolution image blocks; K is a preset value; Step 2.3, according to the current reconstructed high-resolution image block obtained in step 2.2, return to repeat step 2.2 until the number of iterations reaches the preset value; Step 3, superimpose the corresponding reconstructed high-resolution image blocks of all image blocks in the input low-resolution face image according to the position, and then divide by the number of overlaps of each pixel position to reconstruct a high-resolution face image. 2. according to claim 1, based on the human face magic construction method of local constraint iterative neighborhood embedding, it is characterized in that: Let the image block set obtained by dividing the input low-resolution face image X ^t be

It is estimated that the image block set obtained by dividing the high-resolution face image Y ^t (0) is

Divide all high-resolution face sample images in the high-resolution training set to obtain a high-resolution image block set Y={y _ij |1≤i≤N, 1≤j≤M}, and for all low-resolution training sets The high-resolution face sample images are respectively divided to obtain a low-resolution image block set X={x _ij |1≤i≤N, 1≤j≤M}; where, the identifier i represents the high-resolution face sample in the high-resolution training set The serial number of the image and the serial number of the low-resolution face sample image in the low-resolution training set, and the mark j represents the serial number of the block position on the image;

is the image block at position j of the estimated high-resolution face image Y ^t (1),

is the image block at position j of the input low-resolution face image X ^t ; y _ij is the image block at position j of the i-th image in the high-resolution training set, x _ij is the position of the i-th image in the low-resolution training set The image block at j; the number of low-resolution face sample images in the low-resolution training set and the number of high-resolution face sample images in the high-resolution training set are recorded as N, and M is divided into image blocks for each image the number of blocks; In step 2, any image block in the input low-resolution face image

Perform the following sub-steps, Step 2.1, let the initial value of the number of iterations p be 1; for an image block at any position in the input low-resolution face image

Establish the low-resolution face sample block space X _j ={x _ij |1≤i≤N} as the low-resolution image block dictionary, and establish the high-resolution face sample block space Y _j ={y _ij |1≤i≤ N} as a high-resolution image block dictionary, looking for the corresponding position to estimate the high-resolution image block

in ${\{\mathrm{the\; y}}_j^t\left(1\right)=\mathrm{Bicubic}\left(x_j^t\right)|1\&le;j\&le;m\};$ Step 2.2, using the reconstructed high-resolution image block obtained by performing step 2.2 in the previous iteration, to calculate the reconstructed high-resolution image block of this iteration, including the following sub-steps: Step 2.2.1, according to the current reconstructed high-resolution image patch Calculate the distance from each image block in the high-resolution image block dictionary Y _j ={y _ij |1≤i≤N} at the corresponding position, and find the K image blocks with the smallest distance as follows, $\mathrm{<span\; class="notranslate">\; dist</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; N</span>}$ ${\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; support</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; dist</span>}<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; )</span>$ Among them, dist(p)∈R ^N , dist(p) represents the reconstructed high-resolution image block

The distance from each image block y _ij in the high-resolution image block dictionary, R ^N represents the N-dimensional real number space; dist| _K represents the smallest K values in dist(p), is the current high-resolution image patch

An index set of K image blocks with the smallest distance in the high-resolution image block dictionary; When step 2.2.1 is performed for the first time, the current reconstructed high-resolution image patch

Using estimated high-resolution image patches After performing step 2.2.1, the current reconstructed high-resolution image block

The reconstructed high-resolution image block obtained by performing step 2.2.3 during the previous iteration; Step 2.2.2, find the set obtained in step 2.2.1

The corresponding image blocks x _k of the K image blocks y _k in the low-resolution image block dictionary, and the image blocks

Perform linear reconstruction to obtain the optimal weight coefficient

as follows, in, Returns the value of w(p) when the function about the weight coefficient w(p) obtains the minimum value, w _k (p) represents the component corresponding to x _k in the weight coefficient w(p), τ is the balance reconstruction error and Locally constrained regularization parameters; In step 2.2.3, the reconstructed high-resolution image block of this iteration is reconstructed by the following formula, ${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; K</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; w</span>}}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}$ in,

is the optimal weight coefficient obtained in step 2.2.2

The component corresponding to the image block y _k in ; Step 2.3, judge whether the number of iterations reaches the preset value, otherwise set p=p+1, return to repeat step 2.2 for the next iteration according to the reconstructed high-resolution image block obtained in step 2.2.3 in this iteration; In this iteration, the reconstructed high-resolution image block obtained in step 2.2.3 is used as the final reconstructed high-resolution image block, and the iteration ends.

Images