CN101930543B - Method for adjusting eye image in self-photographed video - Google Patents

Abstract

The invention discloses a method for correcting an eye image in a selfie video, comprising the following steps: step 1, target eye image detection and positioning: detecting and locating the position of the eye from the video image; step 2, the sclera image in the eye image, Recognition and positioning of iris image and pupil image: recognize sclera image and iris image according to grayscale; recognize iris image and pupil image according to texture; locate the relative position of sclera image and iris image, iris image and pupil image; Step 3, iris The secondary projection of the image and the pupil image translates the iris image and the pupil image to the center of the sclera image, thereby realizing the adjustment of the eye image. The present invention performs image processing through a software method without adding additional devices, so that when a person's face is facing the display device and the eyes are not watching the camera lens, the moving video image of the eyes gazing at the display device can be obtained on the display device, which is greatly improved. The improvement cost of the hardware system is reduced.

Description

A method for adjusting eye images in selfie videos

technical field

The invention relates to the field of video data processing and imaging, in particular to a method for adjusting eye images in selfie videos.

Background technique

In the application field of digital video processing, especially with the popularization of 3G communication network, the application of video Selfie and network video becomes more and more extensive. At present, there is a very disturbing situation, that is, video capture devices are generally located near the outer edge of the display device, as shown in FIG. 2 . In this case, when the person being photographed looks at the screen of the display device, the imaging result on the display device is that the gaze point of the eyes deviates from the screen of the display device. In short, it is the face display image obtained by the screen observer The image of the middle eye is crooked, not facing up. The "face up" and "non-face up" of the eye image perceived by human vision are mainly based on the relative positions of the sclera, iris and pupil in the human eye. It is generally believed that the iris and pupil are located in the sclera. The central position is "face-to-face", otherwise it is "non-face-to-face". And when the person being photographed gazes at the video capture device, the imaging result on the display device is that the gaze point of the eyes faces the screen of the display device, but the person being photographed cannot see this imaging result, as shown in Fig. 1a and Fig. 1b.

Contents of the invention

Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for correcting the eye image in the self-portrait video, so that when the subject is watching the screen, the video acquisition device collects and finally displays is the image that the eyes face up to.

In order to solve the above-mentioned technical problems, the present invention discloses a method for correcting eye images in a selfie video, comprising the following steps:

Step 1, target eye image detection and positioning: detect and locate the position of the eye from the video image;

Step 2, recognition and positioning of the sclera image, iris image and pupil image in the eye image: distinguish the sclera image and the iris image according to the gray scale; distinguish the iris image and the pupil image according to the texture; locate the sclera image and the iris image, the iris image and the pupil the relative position of the image;

Step 3, the secondary projection of the iris image and the pupil image, which translates the iris image and the pupil image to the center of the sclera image, thereby realizing the adjustment of the eye image.

In the present invention, preferably, said step 1 includes the following steps:

Step (11), preprocessing the image of the selfie video; including using the erosion-expansion method to strengthen the connectivity of each scattered point in the image, and using the median filter to process the image to make the image smoother. This step can adopt common image processing methods in the field. At the same time, this step is not a necessary step of the present invention, but only one of the optimization steps. The present invention can still achieve the purpose of the invention without this step.

Step (12), the image is converted into chromaticity space. Since the skin color of different races does not change much on the two-color difference or hue-saturation plane, the difference of skin color is more in brightness than chroma, so it can be based on Skin color conditions Recognize face images from selfie video images; for example, in the case of good lighting and appropriate contrast, that is, the average brightness value is between 100 and 200, the contrast is between 50% and 80%, and the skin color area is YCbCr The space occupies the range of 102<Cb<128, 125<Cr<160.

Step (13), according to the grayscale method, identify the images of the left and right eyes from the face image; The boundary between the two can be quickly divided according to the difference in grayscale.

In the present invention, preferably, said step 2 includes the following steps:

Step (21), black-and-white binary processing is carried out to the identified eye image, and the sclera image and the iris image are identified according to the grayscale method; After black and white binarization processing, the boundary between the two can be quickly divided according to the difference in gray level.

Step (22), identify the iris image and the pupil image according to the texture analysis method, and calculate the relative position of the iris image and the pupil image; the iris area has more complex textures, while the pupil area basically presents a single texture and the iris area always presents Circle, so block Fourier transform analysis or block discrete cosine transform can be performed on this area. By analyzing the high-frequency components in the transform domain, more high-frequency components indicate that the texture of the area is complex, which is the iris area, and vice versa is the pupil area. , thus giving the boundary between the two in the space domain.

Step (23), calculating the azimuth α and the distance d between the central point of the pupil image and the central point of the iris.

In the present invention, preferably, said step three includes the following steps:

Step (31), the iris image is translated to the center of the sclera image;

Step (32), for the image missing part on the sclera image after the translation of the iris image, use the sclera image around the iris image before translation to fill;

Step (33), according to the azimuth α and the distance d between the central point of the pupil image and the central point of the iris, the circular area where the iris image after translation is located is rotated with the center of the circle as the center; the rotation direction is π+α, and the rotation The angle is rtan ^-1 (d/r), where r is the radius of the pupil.

Step (34), for the vacant part of the iris image after the directional rotation, use the iris image around the sclera image to fill.

The principle of the present invention is that when the subject's line of sight is aligned with the display screen, after detecting the human eye target in the captured video image, the distance between the line of sight and the center of the pupil to the optical center of the camera is judged according to the distribution of the pupil on the eyeball. According to the angle, the area near the eyes of the collected video image is re-projected, and finally the video image with eyes on the screen is displayed on the display screen.

Beneficial effects: the present invention performs image processing through a software method without adding additional devices, so that when the human face faces the display device and the eyes do not watch the camera lens, the active video of the eyes watching the display device can be obtained on the display device image, greatly reducing the improvement cost of the hardware system. The method of the invention has important application prospects in video communication, video conferencing and other aspects that need to use video for two-way or multi-way communication.

Description of drawings

The advantages of the above and/or other aspects of the present invention will become clearer as the present invention will be further described in detail in conjunction with the accompanying drawings and specific embodiments.

Figure 1 is a schematic diagram of gaze alignment and gaze misalignment in reality.

Fig. 2 is a schematic diagram of a common video selfie device in the prior art.

Fig. 3 is a schematic diagram of gaze correction calculation in the present invention.

Fig. 4 is a process diagram of gaze correction calculation in the present invention.

Fig. 5 is a flow chart of the detection of connected regions in the present invention.

Fig. 6 is an example diagram of a Haar-like rectangular feature in the present invention.

Fig. 7 is a simplified flowchart of the method of the present invention.

Detailed ways:

The hardware part of the present invention is composed of a single video shooting device, an arithmetic processing device, and a display device. The core idea is to use the target recognition, target registration and target secondary projection in the video image to realize the video image displayed in the display device by the observer's eyes. .

As shown in Figure 7, the present invention discloses a method for correcting eye images in a selfie video, comprising the following steps:

Step 1, target eye image detection and positioning: detect and locate the position of the eye from the video image;

The first step includes the following steps: step 11, preprocessing the image of the selfie video; step 12, identifying the face image from the image of the selfie video; step 13, identifying the left face image from the face image according to the grayscale method , the image of the right eye.

Step 11, preprocessing the image of the selfie video;

Preprocessing the image of the selfie video, because the image acquisition is often carried out in a changeable and unpredictable environment (mainly the lighting environment), it is particularly necessary to preprocess the image so that it can adapt to the requirements of the algorithm. The image preprocessing involved in the present invention includes histogram equalization, morphological operation and median filtering.

Histogram equalization is the most basic operation in digital image processing, and its function is to make the contrast of the image clear. Morphological operations are divided into morphological erosion and morphological expansion, which are performed on binary images. Corrosion before expansion is called the closing operation, which can close the defective graphics in the image, and the opposite is called the opening operation, which makes the closed image break. After morphological operations, the isolated noise points in the image can be removed and the disconnected connected areas caused by various reasons can be repaired.

Median filtering is a nonlinear signal processing technique that can effectively suppress noise. The basic principle of median filtering is to replace the value of a point in a digital image or digital sequence with the median value of each point in a neighborhood of the point, thereby eliminating isolated noise points. The image will be smoothed after median filtering. Step 12, identify the face image from the image of the selfie video, including:

Face detection based on skin color segmentation:

Most face analysis methods are based on grayscale images, and skin color segmentation uses the color and chromaticity information of human skin color as features for face detection. It is a face detection method based on feature invariants.

There is a clear difference between human skin color and the natural background. Due to the effect of facial blood vessels, the red component is relatively full; and the skin color is relatively maintained within a stable range under different lighting and ethnic conditions. At the same time, this method only needs to traverse the global image several times, which is fast and easy to implement. It is a basic algorithm widely used in face detection systems.

The algorithm is mainly divided into three steps:

Step a, skin color area segmentation: use the YCbCr color space for skin color segmentation, in this space, the threshold of the skin color Cr component is easy to select, and is less affected by light. The conversion relationship between YCbCr and RGB color space is:

Y＝0.256789R+0.504129G+0.0097906B+16

Cb=-0.148223R-0.290992G+0.439215B+128

Cr＝0.439215R-0.367789G-0.071426B+128

R=1.164383×(Y-16)+1.596027×(Cr-128)

G=1.164382×(Y-16)-0.391762×(Cb-128)-0.812969×(Cr-128)

B=1.164382×(Y-16)+2.017230×(Cb-128)

Through threshold segmentation, the YCbCr color image is converted into a black and white image, black represents the background, and white marks the area close to the skin color. In general, that is, when the lighting is good and the contrast is appropriate, the skin color area occupies the range of 102<Cb<128, 125<Cr<160 in the YCbCr space, so the threshold for segmentation can be selected as Cb=116, Cr=144.

In step b, all connected white areas will be located, and some preprocessing is performed on the image before detecting the area value: (1) Use morphological closing operation (corrosion-expansion) to strengthen the connectivity of each scattered point. (2) Use median filtering to smooth the image.

In step c, among all the found white areas, the areas most likely to be human faces are screened out based on information such as area, aspect ratio, position, etc. In this embodiment, the proportion of the face in the image is generally high (for example, more than 60%), the position is in the center of the image, and the aspect ratio is close to 1:1, so it is easy to distinguish.

Detection of connected areas: In the face skin color segmentation algorithm, an important step is to detect the connected areas in the image after the morphological operation, and determine the coordinates, size, aspect ratio, etc. of the smallest rectangular boundary surrounding these areas. . In order to detect, first determine some rules that define "connected regions", these rules can be expressed as: (1) two points are connected to require them to be adjacent in rows or columns (obliquely adjacent are not considered connected); (2) if A connected region contains another connected region, the latter will be ignored; (3) If the smallest rectangular boundary of one connected region partially overlaps with the smallest rectangular boundary of another connected region, the two are defined as independent two connected regions. According to these rules, an algorithm for traversing the boundary is designed to search for connected skin-color-like pixels row by column, and define the skin-color-like pixels as 1 and the non-skin-color pixels as 0. The algorithm flow representation is shown in Figure 5, and Figure 5a and Figure 5b are the overall flow of multiple area detection and the specific flow of single area detection respectively.

As shown in Figure 5a, start from the first pixel in the first row of the upper left corner of the image to traverse line by line, judge whether the current pixel is the starting point, if so, apply the method in Figure 5b to detect the boundary with this pixel as the starting point, and complete After detection, mark the boundary and point to a pixel to the right of the previous starting point; if it is not the starting point, continue to traverse the next pixel by row, and repeat the above process until all pixels are traversed.

Detecting the boundary As shown in Figure 5b, the boundary is detected from the starting point as the current point. First, find whether there are pixels of the same type around the current point in the clockwise direction of left, top, right, and bottom, and then judge whether the found pixel point is a boundary point. For the convenience of description, the present invention assumes that there are pixels of the same type on the left of the current point as an example. Set the current pixel point A, and find out whether there is a pixel point B of the same type on the left side of point A. If not, update the boundary and judge whether it can return to the starting point. If it cannot return to the starting point, then find point A again. Whether there is a pixel point B of the same type on the left, if it can return to the starting point, it will end; if there is a pixel point B of the same type, set B as the current point, and then judge the first counterclockwise direction of the AB direction, that is, "down" If there are no pixels of the same type, update the boundary and find out whether there is a pixel B of the same type on the left side of point A. If there is a pixel C of the same type below, it means that A is not a boundary point. Set the point to C and go down in the counterclockwise direction of BC to find out whether it returns to the starting point. If it returns to the starting point, it ends. If it does not return to the starting point, it judges whether there is a point on the right. Point B is an edge point, update the boundary and continue to search the boundary clockwise with the AB direction as the starting direction, and re-judge whether there is a pixel point below, if there is a point, go to the right to judge whether to return to the starting point, if it returns to the starting point If it does not return to the starting point, then judge whether there is a point on the upper side, if not, update the boundary, and re-judge whether there is a point on the right; if there is a point above, go up, and judge whether to return to the starting point, if yes Then end, if it is not back to the starting point, then re-judgment whether there is a point on the left, until the starting point is found. Through the boundary information, the area and aspect ratio of the region are obtained, and the regions with too large or too small area and too large or too small aspect ratio will be excluded.

This embodiment adopts face detection based on AdaBoost

Adaboost specifically solves two problems: one is how to deal with training samples? In AdaBoost, each sample is assigned a weight. If a sample is not classified correctly, its weight will be increased, otherwise it will be decreased. In this way, the AdaBoost method pays more attention to the "difficult" samples. The second is how to merge weak classifiers into a strong classifier? The strong classifier is expressed as a linear weighted sum of several weak classifiers, and the weight of the weaker learning machine with higher accuracy is higher. Therefore, AdaBoost contains two important ideas, one is the fusion of multiple features (this is the core of the Boosting algorithm), and the other is weighted classification, which assigns multiple features to different weights, and the weights are obtained through training (traditional weighted classification. weights are predictable). Combined with reality, for face detection, to obtain the face from the background, it has been stated that it must be based on certain features, such as texture and edge features, etc. In this embodiment, the Haar-like method is used for feature extraction . Select K features, that is, there are K weak classifiers and T training samples, and the weight combination of K feature vectors with the highest classification accuracy is obtained through loop testing, and the weight values of T training samples are continuously updated during the loop process. Increase the weight of hard-to-classify and reduce the weight of easy-to-classify. This method uses the AdaBoost method for face detection. The details of the algorithm will be elaborated below. All algorithms are based on grayscale images.

Haar-like feature extraction The Haar-like feature is a rectangular pair feature. Given limited data, the detection based on the Haar-like feature can encode the state of a specific area. The rectangular feature is for some simple graphic structures, such as edges and line segments. It is more sensitive, but it can only describe the structure of a specific direction (horizontal, vertical, diagonal), so it is relatively rough. Some features of the face can be simply described by rectangular features. For example, usually, the eyes are darker than the cheeks, which is an edge feature; the sides of the bridge of the nose are darker than the bridge of the nose, which is a linear feature; The surrounding color is darker, etc., as shown in Figure 6b, which is a characteristic of a specific direction. Commonly used feature rectangles are divided into edge features, linear features, and specific direction features. As shown in Figure 6a, edge feature templates are used to extract edge information at different angles, and linear feature templates are used to extract linear image blocks at different angles. Feature templates are used to extract image blocks of a specified type.

The basic template is the smallest in size, so similar templates of various sizes can be formed by scaling, for example, edge feature template 1 is a template with 2 pixels. The feature value of the template when traversing the image is the sum of the areas covered by the white rectangles on the image minus the sum of the areas covered by the black rectangles. So all the features on a solid color image will have zero values. The feature templates of each template can be placed "arbitrarily" with "arbitrary" size in the sub-window, and each form is called a feature. Finding out all the features of the sub-window is the basis for weak classification training.

For an image to be detected, such as an m×n image, obviously it contains a large number of features, and the sum of the total number of multiple features is a considerable number. Therefore, the following will discuss the problem of the total number of features contained in the image. Visually speaking, the image is a big box, and the template is a small box that moves freely in it. The small box has many different placement positions in the big box. The sum of all possible placement positions of small boxes of various scales is the feature total. Let the size of the template be s×t, then the total number of features contained in m×n for:

${\mathrm{<span\; class="notranslate">\; \&Omega;</span>}}_{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; pq</span>}$

$<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\frac{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; \&times;</span>\frac{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}$

$<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\frac{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}<span\; class="notranslate">\; \&times;</span>\underset{\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\frac{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; t</span>}}$

The sum of the number of features of various templates is the total number of features in the image. Usually, for 2 edge templates, 2 linear templates, 1 specific direction template, and 5 templates, the number of features in a 16×16 size image is 32384 , if the image size is 36×36, the number of features will reach 816264.

Integral graph operation From the above data, it can be seen that the number of features in an image is very large, and increases sharply with the size of the image. Therefore, it is necessary to find a suitable feature calculation method. The integral graph method adopted in this embodiment is an effective and fast feature calculation method.

For an image A, the integral image value of point A(x, y) is defined as:

$\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>$

That is, the point and the origin are the sum of all points in the rectangle whose corners are opposite. The Haar-like rectangular feature of the image can be calculated quickly and conveniently by using the integral image. The calculation of the eigenvalue of the rectangular feature is only related to the integral map of the endpoint of this feature, and has nothing to do with the image coordinate value. Therefore, regardless of the scale of the rectangular feature, the calculation of the feature value takes a constant time, and it is just a simple addition and subtraction. Because of this, the introduction of the integral map greatly improves the detection speed.

AdaBoost design process AdaBoost algorithm is ultimately to obtain a suitable strong classifier. The process of designing a classifier is mainly a training process. Training uses a large number of samples, including human faces and non-human faces. The process is as follows:

1) Given a series of training samples (x ₁ , y ₁ ), (x ₂ , y ₂ ),,, (x _n , y _n ), where y _i =0 means it is a negative sample (non-face), y _i =1 means it is a positive sample (face). n is the total number of training samples;

或

其中m正样本的数量，l为附样本的数量，m+l＝n；2) Initialize the weight W _{1, i} = D(i), let

or

Where m is the number of positive samples, l is the number of additional samples, m+l=n;

3) For t=1...T first normalize the weight, T is the number of iterations:

${\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}}$

Then for each feature f, train a weak classifier h(x, f, p, θ); calculate the weighted q _t error rate ε _f of the weak classifier corresponding to all features, where f is the feature, θ is the threshold and p indicates the direction of the inequality sign:

ε _f ＝∑ _i q _i |h(x _i , f, p, θ)-y _i |

Then select the best weak classifier h(x) (with the minimum error rate ε _t ):

ε _t = min _{f, p, θ} ∑ _i q _i |h(x _i , f, p, θ)-y _i |

=∑ _i q _i |h(x _i , f _t , p _t , θ _t )-y _i |=∑ _i q _i |h _t (x)-y _i |

The training and selection of weak classifiers will be described in detail below. According to this best weak classifier, adjust the weights:

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}$

where e _i = 0 means that x _i is correctly classified, where e _i = 1 means that x _i is incorrectly classified;

${\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}$

The final strong classifier is:

in

${\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}$

A weak classifier h(x, f, p, θ) consists of a feature f, a threshold θ, and p indicating the direction of the inequality:

For the rectangular feature in this embodiment, the feature value f(x) of the weak classifier is the feature value of the rectangular feature. Since the size of the selected training sample set is equal to the size of the detection sub-window during training, and the size of the detection sub-window determines the number of rectangular features, so the features of each sample in the training sample set are the same and the number is the same, and a feature There is a fixed eigenvalue for a sample. For an ideal image with randomly distributed pixel values, the average of the feature values of the same rectangular feature to different images should tend to a constant value K. This situation should also occur on non-face samples, but since non-face samples are not necessarily images with random pixels, the above judgment will have a large deviation. For each feature, calculate the average value of its feature values for all types of samples (face or non-face), and finally get the average distribution of all features for all types of samples. There is not much difference between the distribution curves of face samples and non-face samples, but after the feature value is greater or smaller than a certain value, there is a consistent difference in the distribution curve, which shows that most of the features are useful for recognizing faces and non-face samples. The ability of the face is very small, but there are some features and corresponding thresholds that can effectively distinguish between face samples and non-face samples.

The requirement of a weak learner (a feature) is only: it can distinguish between face and non-face images with an error rate slightly lower than 50%, so the above mentioned can only be accurately distinguished within a certain probability range It is totally enough. According to this requirement, all rectangular features with an error rate lower than 50% can be found (appropriately choose the threshold, for a fixed training set, almost all rectangular features can meet the above requirements). In each round of training, the best weak classifier in the current round will be selected (in the algorithm, iterating T times is to select T best weak classifiers), and finally the best weak classifier obtained in each round will be improved according to a certain method is a strong classifier.

Training a weak classifier (feature f) is to determine the optimal threshold of f under the current weight distribution, so that the weak classifier (feature f) has the lowest classification error for all training samples. Selecting an optimal weak classifier is to select the weak classifier (feature) whose classification error for all training samples is the lowest among all weak classifiers. For each feature f, the feature values of all training samples are calculated and sorted. By scanning the sorted feature values once, an optimal threshold can be determined for this feature, thus training a weak classifier. Specifically, for each element in the sorted list, the following four values are computed:

1) the sum T ⁺ of the weights of all face samples;

2) The sum T- of the weights of all non-face samples;

3) The sum S+ of the weights of the face samples before this element;

4) The sum S- of the weights of the non-face samples before this element;

和它前面的一个特征值之间的数作为阈值时，所得到的弱分类器就在当前元素处把样本分开——也就是说这个阈值对应的弱分类器将当前元素前的所有元素分类为人脸(或非人脸)，而把当前元素后(含)的所有元素分类为非人脸(或人脸)。In this way, when selecting the feature value of the current element

and an eigenvalue preceding it When the number between is used as the threshold, the obtained weak classifier will separate the samples at the current element—that is to say, the weak classifier corresponding to this threshold will classify all elements before the current element as faces (or non-faces) , and classify all elements after (including) the current element as non-human faces (or human faces).

It can be considered that the classification error brought by this threshold is:

e=min(S ⁺ +(T ^- -S ^- ), S ^- +(T ⁺ -S ⁺ ))

By scanning the sorted table from the beginning to the end, the threshold (optimum threshold) that minimizes the classification error can be selected for the weak classifier, that is, an optimal weak classifier is selected.

The AdaBoost strong classifier is formed by cascading weak classifiers. When the strong classifier treats an image to be detected, it is equivalent to asking all the weak classifiers to vote, and then the voting results are weighted and summed according to the error rate of the weak classifiers. The weighted sum result is compared with the average voting result to obtain the final result. The average voting result, that is, assuming that all weak classifiers have the same probability of voting "yes" and "no" votes, the average probability obtained is:

$\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}$

Step 13: Identify images of left and right eyes from the face image according to the grayscale method.

Step 2, recognition and positioning of the sclera image, iris image and pupil image in the eye image: recognize the sclera image and iris image according to the gray scale; recognize the iris image and the pupil image according to the texture; locate the sclera image and iris image, the iris image and the pupil the relative position of the image;

Described step 2 comprises the following steps: Step 21, carries out black-and-white binarization processing to the identified eye image, and recognizes the sclera image and the iris image according to the difference in gray scale, binarization adopts the ostu method, promptly selects to make binary value The image grayscale variance and the maximum threshold are binarized. Due to the large difference in grayscale between the sclera image and the iris image, the sclera area is white and the iris area is black after binarization. According to the iris area boundary is a circle The features of the shape can easily distinguish the two; step 22, identify the iris image and the pupil image according to the texture analysis method, and calculate the relative position of the iris image and the pupil image; the iris area has more complex textures, and The pupil area basically presents a single texture and the iris area is always circular, so block Fourier transform analysis or block discrete cosine transform can be performed on this area. By analyzing the high-frequency components in the transform domain, more high-frequency components indicate this area The texture is complex for the iris area and vice versa for the pupil area, thus giving the boundary between the two in the spatial domain. In the present invention, it is determined by comparing the ratio of high-frequency components to low-frequency components between two image spectrums. During actual calculation, it can generally be considered that when the frequency spectrum is bimodal, the high-frequency components account for more than 20% of the total spectrum energy and can be defined as iris area. Step 23, calculating the azimuth and distance between the central point of the pupil image and the central point of the iris. Since both the pupil and the iris are circular, the acquisition of the center point of the pupil and the center point of the iris in the present invention is obtained by extracting the arc boundaries of the pupil and iris respectively, and positioning them by using the geometric relationship between the center of the circle and the arc.

Step 3, secondary projection of the iris image and the pupil image, moving the iris image and the pupil image to the center of the sclera image through directional rotation, so as to realize the adjustment of the eye image.

As shown in Figure 4, the oblique striped area represents the vacant part after translation and directional rotation, the dot matrix part represents the iris, and the black area represents the pupil. Said step three comprises the following steps: Step 31, translate the iris image to the center of the sclera image, as shown in Figure 4 (a); Step 32, for the image missing part on the sclera image after iris image translation, use the iris before translation The sclera image around the image is filled, as shown in Figure 4 (c); Step 33, according to the azimuth and distance from the center point of the pupil image to the center point of the iris, the circular area where the iris image after translation is located is taken as the center of the circle The center undergoes a directional rotation, as shown in Fig. 3 and Fig. 4(c). According to the azimuth α and distance d between the center point of the pupil image and the center point of the iris, the circular area where the translated iris image is located is rotated with the center of the circle as the center; the rotation direction is π+α, and the rotation angle is r tan ^{- 1} (d/r), where r is the radius of the pupil; step 34, for the vacant part of the iris image after the direction rotation, use the iris image around the sclera image to fill.

The preprocessing in step 11 includes using the erosion-dilation method to strengthen the connectivity of each scattered point in the image. The preprocessing in step 11 includes using histogram equalization to improve the contrast of the image. The preprocessing in step 11 includes processing the image using a median filter.

The present invention provides an idea and method of a method for correcting eye images in a selfie video. There are many methods and approaches for realizing the technical solution. The above description is only a preferred embodiment of the present invention. Those of ordinary skill in the art can also make some improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components that are not specified in this embodiment can be realized by existing technologies.

Claims (5)

1. the regulation method of an eye image in shooting the video is characterized in that, may further comprise the steps:

Step 1, subject eye image detection and location: the position of from video image, detecting and locate eyes;

Step 2, the differentiation of sclera image, iris image and pupil image and location in the eye image: tell sclera image and iris image according to gray area; Tell iris image and pupil image according to texture area; The relative position of location sclera image and iris image, iris image and pupil image; Saidly tell iris image and pupil image method for this iris region and pupil region are carried out analysis of piecemeal Fourier transform or piecemeal discrete cosine transform according to texture area; Through analytic transformation territory medium-high frequency component; Bright this zone-texture of high fdrequency component multilist is complicated; Be iris region, otherwise then be pupil region, thereby provide spatial domain boundary between the two;

Step 3, the reprojection of iris image and pupil image moves to the center of sclera image with iris image and pupil image through oriented rotation, thereby realizes the levelling of eye image;

Said step 3 may further comprise the steps:

Step (31) moves to iris image at the center of sclera image;

Step (32) for the disappearance of the image on the sclera image after iris image translation part, uses the preceding iris image of translation sclera image on every side to fill;

Step (33) according to the position angle and the distance of pupil image central point distance iris central point, is carried out the oriented iris image center that moves to pupil image;

Step (34) to the vacancy part that pupil image is oriented after moving, uses iris image to fill.

2. a kind of regulation method from the middle eye image that shoots the video according to claim 1 is characterized in that said step 1 may further comprise the steps:

Step (11) is carried out pre-service to the image that shoots the video certainly;

Step (12) identifies facial image from the image that shoots the video certainly;

Step (13) identifies the image of left and right eyes from facial image according to gray-scale relation.

3. a kind of regulation method from the middle eye image that shoots the video according to claim 2 is characterized in that said step 2 may further comprise the steps:

Step (21) is carried out the black and white binary conversion treatment to the eye image that identifies, and is identified sclera image and iris image according to gray-scale relation;

Step (22) identifies iris image and pupil image according to texture analysis method, and calculates the relative position of iris image and pupil image;

Step (23) calculates the position angle and the distance of pupil image central point distance iris central point.

4. a kind of regulation method from the middle eye image that shoots the video according to claim 2 is characterized in that the pre-service of said step (11) comprises the connectedness of using the corrosion plavini to strengthen each spaced point in the image.

5. a kind of regulation method from the middle eye image that shoots the video according to claim 2 is characterized in that, the pre-service of said step (11) comprises uses medium filtering to handle image.

Images