CN101561710B - A Human-Computer Interaction Method Based on Face Pose Estimation - Google Patents

Abstract

The present invention relates to a human-computer interaction device and method based on face pose estimation. The steps of realization are as follows: the face image sequence is obtained through a camera, and five features of two eye corners, two mouth corners and nose tip are automatically extracted after preprocessing point; using a frontal image as a reference, estimate the three deflection angles of the face in any image according to the position and corresponding relationship of the five feature points; define the position and operation mode of the mouse pointer by these posture information, and generate human-computer interaction information; Connect to the computer through the USB interface to form a new visual mouse device. As a supplement to traditional human-computer interaction methods, the device is suitable for some special interactive groups (such as handicapped people) and interactive environments (such as multimedia games), and has significant application value.

Description

A Human-Computer Interaction Method Based on Face Pose Estimation

technical field

The present invention relates to a human-computer interaction method, in particular to a method for constructing a visual mouse device based on facial feature point positioning and attitude detection. Two corners of the eyes, two corners of the mouth and There are 5 feature points on the tip of the nose. Taking the position of the corresponding feature points of a frontal image as a reference, the three deflection angles of the real-time face image are estimated, and human-computer interaction information is generated, which is connected to the computer through the USB interface to form a human-computer interface. A visual mouse device for computer interaction.

Background technique

Human-computer interaction technology (Human-Computer Interaction Techniques) refers to the technology that realizes the dialogue between human and computer in an effective way through computer input and output devices. It includes that the machine provides a large amount of relevant information and prompts to people through output or display devices, and people input relevant information to the machine through input devices to answer questions, etc. Human-computer interaction technology is one of the important contents in computer user interface design. It is closely related to the subject areas of cognition, ergonomics, psychology and so on.

In the field of human-computer interaction, traditional devices such as keyboards, mice, and light pens are widely used. However, traditional devices have their limitations in some special cases, such as the complex interface operations of some multimedia games and the use of computers by disabled people. Design input devices that do not rely on body movements as a supplement to existing human-computer interaction methods.

In order to adapt to different interaction environments and apply to different interaction groups, domestic and foreign scholars have extensively studied interaction methods based on vision (image), hearing (voice), and touch (pressure, temperature), and some methods have been initially put into use. Among the visual interaction methods, there are currently research hotspots such as eye-tracking technology and sign language recognition technology. Gaze tracking technology perceives people's attention point of view through the camera and pupil positioning technology, so as to drive mouse positioning and related operations. However, there are two important shortcomings: one is that the pupil movement is discontinuous and cannot be well matched with the continuous moving point of view. The accuracy of tracking is reduced; the second is that due to physiological and psychological reasons, the movement of the human eye is random, and in some cases the position of the human eye's line of sight does not reflect the subjective will of its own attention.

In order to overcome the defects of the human-computer interaction method of line of sight tracking, the present invention adopts a face pose estimation method, uses a video camera to obtain a face image, realizes face detection and key point positioning through image and video processing technology, and according to the characteristics of multi-frame sequence images It is a new interaction method to estimate the face pose based on the point position distribution, and realize human-computer interaction based on the pose information. The resulting human-computer interaction device is suitable for some special interactive groups (such as the disabled) and interactive environments (such as multimedia games), has significant application value, and has great significance in the field of human-computer interaction.

Contents of the invention

Aiming at the fact that existing human-computer interaction devices cannot completely satisfy different interactive environments and interactive groups, the purpose of the present invention is to provide a human-computer interaction method based on face pose estimation, which utilizes facial feature point automatic positioning and human face Posture estimation and other technologies generate human-computer interaction information, and connect to a computer through a USB interface to form a human-computer interaction visual mouse device.

The present invention involves comprising the following steps:

a) The face digital image sequence obtained through the optical lens and CMOS image sensor components, and the data acquisition is realized through the digital video high-speed channel of the DSP.

b) Preprocessing the face digital image.

① Image denoising. Since image noise points are significantly different from their neighboring pixels in gray value, statistical characteristics or distribution rules, image filtering can be performed to achieve noise suppression, making useful information easier to detect and identify. Nonlinear diffusion is a good image denoising method, which has good edge-preserving properties, but it needs to go through many iterations, and the computational complexity is high. In the present invention, the edge mapping is constructed by using the variance of the strip area in four directions of the target point, which is applied to the non-linear diffusion image denoising algorithm, reduces the number of iterations, and uses the integral map to realize fast calculation.

②Face detection and area division. Manually mark multiple faces, transform these faces to the same size, and then calculate the average to get the face template, use the template matching method to get the approximate position of the face, and then divide the face into left and right according to the prior knowledge of face structure features Eyes, nose and mouth areas.

③Edge detection. Since the five feature points are located on the corners of the face organs, it is necessary to extract contour or corner information to improve the positioning accuracy. Traditional edge detection methods include Canny operator, Sobel operator, morphological method and wavelet transform method. In addition, the SUSAN corner detection algorithm can well reflect the edge and corner information, but the effect is affected by the threshold. In the present invention, a direction filtering method is used to obtain edge images.

c) Locate the 5 feature points of the face (2 corners of the eyes, 2 corners of the mouth and the tip of the nose).

① Locate the corner of the eye. Firstly, within the previously determined human eye area, grayscale projection is performed on the denoised image in both horizontal and vertical directions, and the position of the valley point is taken as the two-dimensional coordinates of the eyeball center, and a rectangular area is defined as the eye. The exact position of ; Then, examine this area, binarize the edge image, and detect the number of points in any connected area, and remove the area with too few connected points. Finally, the leftmost (right) point is extracted from the selected connected area of the human eye as the corner point.

② Locate the corners of the mouth. Mouth corner positioning is also corner detection, the method is similar to eye corner positioning, but because the gray scale change of the mouth does not have high-contrast pupils like eyes, resulting in unclear outlines. Firstly, a Gaussian mixture probability model is established by using lip color sample training, and all points in the mouth area are substituted into the model to calculate their probability, and normalized as new grayscale information, and then the mouth corner position is extracted using the same method as eye corner positioning.

③ Locate the tip of the nose. The tip of the nose is not on the contour, and the positioning of the tip of the nose depends on its geometric relationship with the corners of the eyes and mouth. Since the tip of the nose is close to the nostrils and above the line connecting the two nostrils, the grayscale projection method is used to search for the positions of the two nostrils in the nose area, and then the high light point above the nostrils is used as the position of the nose tip. When the deflection angle of the human face is large, the positioning of the tip of the nose is likely to cause errors, but in the present invention, the position of the tip of the nose only affects the sign of the deflection angle, that is, determines the direction of rotation, and does not affect the calculation of the angle, so it is not sensitive to errors.

d) Estimate the 3 deflection angles of the face

The three deflection angles are the basis for generating human-computer interaction information. Under the condition that the deflection angle is not too large, the size of the face is relatively fixed and is in the center of the screen, and there is good light, 5 feature points can be obtained accurately, and 3 deflection angles can be obtained by the following steps.

① Coordinate transformation. First calculate the center of the left and right eye corners, and use it as the origin of the image, and perform coordinate transformation on all other coordinate points according to their relative positions with the origin to obtain new coordinate values. Then calculate and record the center positions of the two corner points of the mouth, and obtain five feature points of the two eye corners, the center of the eye corners, the tip of the nose, and the center of the mouth corners. These five feature points are the basis for estimating the three-dimensional deflection angles of the face.

② Determine a frontal image, take the line connecting the two corners of the eyes as the horizontal line, and the plane defined by the line and the center point of the mouth corner is the front when it is perpendicular to the normal of the camera lens, and set the three deflection angles at this time to 0. Locate the 5 feature points of the frontal image, complete the coordinate transformation according to the method d)①, calculate and record the coordinates of the 5 feature points, and use them as reference information in the subsequent pose estimation.

③ For the face image collected at any time, it is necessary to estimate its pose to generate human-computer interaction information. First locate 5 feature points, and then obtain the coordinates of 5 points according to the same method as d)②. Referring to the feature points of the frontal image obtained in d)②, these coordinates are verified according to the geometric constraint relationship, and the positioning results that obviously do not meet the conditions are discarded, and no interactive information is generated.

④ Use the coordinates of the two images corresponding to 5 points determined in the two steps of d) ② and d) ③, use the pinhole camera model, make some simplified assumptions that are reasonable and do not affect the results according to the characteristics of the algorithm, and finally use the principle of epipolar geometry Derivation is performed to calculate the pose of the face in any image and obtain three deflection angles.

e) Generate human-computer interaction information

The mouse position and operation mode are defined by the three deflection angles obtained earlier. The computer-mouse interaction method includes operations such as arbitrary movement of the pointer in the two-dimensional plane, single and double clicks of the left and right keys of the mouse, and the like. The invention locates the position of the mouse pointer according to the two angle values, and defines the operation of the mouse by the inter-frame mutation amount of the other angle, so as to generate human-computer interaction information.

f) Connect to a computer for communication

The USB interface is developed in the device, the driver program is written according to the communication protocol of the standard USB mouse, and the human-computer interaction information obtained earlier is transmitted to the computer. The computer does not need any special program support, which reduces the burden on the target computer and does not affect the operator's use of the computer to run complex software.

As a supplement to the traditional human-computer interaction method, the present invention is applicable to some special interactive groups (such as handicapped persons) and interactive environments (such as multimedia games), and has significant application value.

Description of drawings

Fig. 1 is the flow chart of human-computer interaction information processing of the present invention

Figure 2 is the face template image used for the initial positioning of the face

Figure 3 is the frontal face organ template and the position map of the five feature points in this method

Fig. 4 is a schematic diagram of the definition method of the three deflection angles of the human face in the present invention

Detailed ways

Below in conjunction with a non-limiting example the present invention will be further described

See Figure 1, Figure 2, Figure 3, Figure 4.

The image acquisition, timing generation and control of the present invention are realized by CPLD devices, the related algorithms of image preprocessing, feature point positioning and attitude estimation are completed by Da Vinci processor TMS32C6446 of TI Company, and the USB interface is realized by Cypress control chip. Complete the design of all algorithms and hardware modules according to the information processing flow, and simulate mouse operation to realize human-computer interaction.

The main modules are introduced as follows:

(1) Face image preprocessing

I. Image denoising. Due to the influence of the optical system or electronic devices, the image will inevitably be disturbed by noise, and denoising processing is required to improve the positioning accuracy of feature points. The principle of nonlinear diffusion denoising algorithm is applied, and the variance value of the gray value of the band-shaped area in the four directions of the neighborhood of the target point is used as the edge map of the image. The diffusion amount of each pixel is determined by the difference The value and the corresponding direction weight coefficient are determined, thereby enhancing the adaptability, reducing the number of iterations, and speeding up the calculation. Its iteration formula is:

${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; )</span>{<span\; class="notranslate">\; \&dtri;</span>}_{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>$

In order not to produce new extremum points in the process of overlapping bands, λ is set to 0.125, and σ _{p, q} represent the variance of banded areas in four directions according to the different values of p and q. The gradient computed for the fork method. The spread function g(σ) is defined as:

$\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; K</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

II. Face localization and area division. Since the present invention is applied to a special human-computer interaction environment, a series of objective conditions can be guaranteed: good illumination, relatively uniform size of the face and being in the center of the image, less influence of the background, and reduced burden of face positioning. Firstly, 100 faces are marked manually, and the face template is obtained by averaging after size normalization, and the position of the face is extracted by template matching method. Let a _{i, j} be the gray level of the pixel to be detected, t _{i, j} be the gray level of the template pixel, and E _a and E _t be the mean values of the two respectively, then the matching coefficient is defined as:

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}{\sqrt{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}$

m>0.55 is the candidate face area. If there are multiple qualified areas within the detection range, the face position is obtained by averaging the continuously appearing face areas. Then the face is divided into four regions: left and right eyes, nose and mouth according to the prior knowledge of face structure features.

III. Edge detection. The eyes, nose, mouth and other organs of the human face contain significant contour information, which can be extracted through edge detection algorithms to lay the foundation for feature point detection. In the present invention, the direction filtering method is adopted: first calculate the gradient size and direction of each point by difference method, then carry out vector superposition to the gradient of the target point and its adjacent 8 points, and use its modulus value as the new gray value of the point , so as to obtain the edge image.

(2) Locate the 5 feature points of the face (2 corners of the eyes, 2 corners of the mouth and the tip of the nose).

I. Locate the corner of the eye. First, for the preprocessed image obtained in (1)I and the general area of the human eye determined in (1)II, horizontal and vertical grayscale projections are performed, and the projected waveform is first smoothed, and then the position of the valley point is taken as the center point of the eyeball. Vertical and horizontal coordinates, and delineate the rectangular area as the position of eyes with this; Secondly, adopt the maximum inter-class variance method to determine the threshold value and carry out binarization processing on the (1) III edge image of this area; Again, carry out binary image Connected area detection, remove the area with less than 20 connected points as interference, and verify the shape of the candidate connected area; finally, extract the leftmost (right) point from the selected human eye connected area, if there are multiple points, the bottom point as the corner of the eye.

II. Locate the corners of the mouth. The method is similar to that of eye corner positioning, but because the grayscale change of the mouth is not as obvious as the eyes, so before the edge detection of the lip area, the skin color transformation is performed through Gaussian mixture models (GMMs): manually collect a large number of lip samples, according to their color First calculate the Cr and Cb color difference information of all sample points, and use this as two-dimensional coordinates to establish a Gaussian mixture probability model composed of two Gaussian models, and substitute all sample points for skin color training to obtain model parameters. Substitute the color information of all points in the mouth area determined in step (1) II into the model, calculate the probability of belonging to the lips, transform the probability to the 0-255 interval as a new grayscale image, and then use the method (1) III for edge detection . The subsequent mouth corner localization steps are the same as eye corner extraction.

III. Locate the tip of the nose. First locate the positions of the two nostrils, and based on the two found corners of the eyes, define the distance as h. According to the positional relationship between the corners of the eyes, the corners of the mouth and the nostrils, a rectangle with a width of h between 1.2h and 1.6h below the line connecting the corners of the eyes Horizontal and vertical grayscale projections were performed in the area, and the valley point position of the horizontal projection was used as the ordinate of the nostril position, and the two valley points of the vertical projection were respectively taken as the abscissa of the two nostrils. Then find the highlight point within 0.3h above the nostril as the position of the tip of the nose.

(3) Estimate the three deflection angles of the face

Under the condition that the deflection angle is not too large (within 20°) and there is good light, the previous steps can accurately locate the 5 feature points A~E, and use the following steps to obtain the 3 deflection angles.

I. Coordinate preprocessing. Calculate the average of the coordinates of A and B to obtain the center point F. Designate F as the coordinate origin, the horizontal axis u to the right, and the vertical axis v to the vertical, and transform the two-dimensional coordinates of all subsequent image points with this as a reference. Average the coordinates of the two mouth corner points D and E to obtain the center point G, together with the two eye corners, the center point of the eye corners and the tip of the nose, a total of 5 feature points A, B, C, F, G, use the following steps to obtain 3 deflection angles .

II. First determine a frontal image, set its three deflection angles α, β, γ to 0, locate 5 feature points A~E, and calculate A, B, C, F, G according to the method (3)I above Point coordinates and record. As a reference image, in the following expressions, a subscript 1 is used as a mark. Then these coordinate points are described as: point A (u _A1 , v _A1 ), point B (u _B1 , v _B1 ), point C (u _C1 , v _C1 ), point F (u _F1 , v _F1 ), point G ( u _G1 , v _G1 ).

III. For the face image collected at any time, first locate 5 feature points, and then obtain the coordinates of 5 points according to the same method as (3)II, and mark 2 as follows: point A (u _A2 , v _A2 ), Point B (u _B2 , v _B2 ), point C (u _C2 , v _C2 ), point F (u _F2 , v _F2 ), and point G (u _G2 , v _G2 ).

IV. Utilize the coordinates of the two images corresponding to 5 points determined by (3)II and (3)III two steps, use the pinhole camera model and the principle of epipolar geometry to perform mathematical derivation, and calculate the posture of the face in image 2, Three deflection angles α, β, γ are obtained. Pinhole camera model:

$\mathrm{<span\; class="notranslate">\; the\; s</span>}\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; K</span>}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; ]</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}$

信息，通过一些合理的假设，在不知道

的情况下估算R中的三个角度值。where s is the scale factor, $\hat{m}={[u,v,1]}^T$ The homogeneous coordinates of the image point, K is the internal parameter matrix of the camera, which can be defined as diag(f, f, 1). T=[t _x , _ty , t _z ] ^T is the translation matrix, $\hat{m}={[x,\mathrm{the\; y},z,1]}^T$ is the homogeneous coordinate of the 3D point of the face in space. R is a rotation matrix based on α, β and γ. The task of this step is based on the five feature points of two known images

information, through some reasonable assumptions, without knowing

Estimate three angle values in R in case of .

Since only angle estimation is involved, the calculation formulas in the algorithm all appear in the form of difference-quotient, and the following assumptions can be made for the parameters of the pinhole camera model: s=1, K=diag(1,1,1), T=[ 0,0,0] ^T . set up:

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; B</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}{{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; B</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}$ $\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; G</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}{{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}$ $\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; B</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}}{{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; B</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}$

c=cosγ d=sinγ h=sinβ

It can be deduced that:

γ=arc tan(M)

$\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arccos</span>}<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; PC</span>}}<span\; class="notranslate">\; )</span>$

$\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arctan</span>}<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; Odd</span>}}{\mathrm{<span\; class="notranslate">\; Ohc</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; hd</span>}}<span\; class="notranslate">\; )</span>$

Since there is a sign problem of β in the calculation process, it cannot be uniquely determined by the four points A, B, F, and G, so point C needs to be introduced, and its sign is determined according to the relative position of the line connecting C, F, and G.

(4) Generate human-computer interaction information

The position and operation mode of the mouse are defined by three angle information of α, β, γ. When α and β are 0, it is defined as the center of the screen. When α changes, the mouse moves up and down, when β changes, the mouse moves left and right, and when the angle is greater than or equal to 20°, the mouse is positioned to the edge of the screen. The mutation of the γ angle defines the mouse operation mode. When γ is positive and the angle change between two consecutive frames is between 3° and 8°, it means clicking the left button, and if it exceeds 8°, it means double-clicking the left button. When γ is negative and the angle changes between two consecutive frames If the angle changes between -3°～-8°, it is a right-click, and if it exceeds -8°, it is a double-click.

(5) Connect to the PC via USB to form a mouse device

The Cypress chip is used to develop the USB interface on the device, and the communication with the PC is realized by means of a standard USB mouse. The mouse position and operation mode generated by the above method are transmitted to the PC instead of the traditional mouse operation information, forming a visual mouse device based on human face pose estimation.

Claims (2)

1. man-machine interaction method of estimating based on human face posture, method may further comprise the steps:

A) people's face digital image sequence that will obtain through optical lens and cmos image sensor assembly is realized data acquisition by the digital video high-speed channel of DSP, and is finished video processnig algorithms by DSP;

B) people's face digital picture is carried out pre-service

At first carry out image noise reduction: adopt the nonlinear diffusion method to realize squelch, wherein edge map adopts the variance computing method of impact point neighborhood four direction belt-like zone; Zone summation in the variance computation process adopts integrogram to realize, to reduce calculated amount; Secondly, the people's face in the image is positioned, and carry out area dividing, obtain the Position Approximate of people's face, according to human face structure feature priori people's face is divided into right and left eyes, nose and four zones of mouth then with template matching method; Carry out rim detection at last, the size and Orientation of examination arbitrary image point gradient, according to the consistance of the gradient direction of 8 of picture point and neighborhoods, with all gradients of 9 ask vector with, as the new gray-scale value of this picture point, obtain edge image with its mould value;

C) inner eye corner of two eyes of people from location face, two corners of the mouths and nose totally 5 unique points:

1. locate the canthus: at first, in the preliminary zone of human eye, adopt the Gray Projection method to determine the eyeball center, and mark the definite position of rectangular area as eyes with this; Then, in this zone, adopt maximum variance between clusters to determine threshold value, edge image is carried out binary conversion treatment, and the restriction connection is counted; At last, extract the most left to the human eye connected region and the rightest o'clock as two canthus points;

2. locate the corners of the mouth: adopt the training of lip colo(u)r atlas to contain the mixed Gauss model of two Gaussian functions earlier, the arbitrfary point belongs to the probability of lip in the calculating face zone, and the normalization back is used the identical method in location, canthus again and is extracted corners of the mouth position as the new half-tone information of this point;

3. locate nose: in nasal area, adopt two naris positions of Gray Projection method search earlier, above the nostril, search highlight then as the nose position;

D) 3 deflection angles of estimation people face

1. coordinate transform: at first, calculate the center at two canthus, all unique points are carried out coordinate transform as image origin; Secondly, calculate the center and the record of two corners of the mouth points, obtain two canthus, center, canthus, nose, corners of the mouth center totally 5 new unique points;

2. to a width of cloth direct picture and there is the realtime graphic of deflection angle to obtain 5 characteristic point coordinates respectively, finish coordinate transform and set up corresponding relation;

3. utilize the coordinate of corresponding 5 of two width of cloth images, derive, calculate the attitude of people's face in the realtime graphic of deflection angle, derive three deflection angles by the pinhole camera modeling of simplifying with to utmost point geometrical principle; Wherein, the pinhole camera modeling of described simplification is that parameter to the pin-hole image machine model makes the following assumptions: scale factor s=1, the inner parameter matrix K=diag of camera (1,1,1), translation matrix T=[0,0,0] ^T

E) generate human-machine interactive information

3. three deflection angles definition mouse positions being derived by step d) and mode of operation are according to two angle values location mouse pointer position wherein, with the operation of the interframe variable quantity definition mouse of another angle, generation human-machine interactive information;

F) connect computer realization communication

Develop USB interface and write driver according to the communication protocol of standard USB mouse, the human-machine interactive information that obtains previously is sent to computing machine, form a kind of new vision mouse apparatus.

2. the man-machine interaction method of estimating based on human face posture according to claim 1, it is characterized in that:, determine that wherein two corners of the mouth points are to participate in attitude and estimate in order to obtain its mid point a width of cloth front reference picture and have the realtime graphic of deflection angle to locate 5 unique points respectively; By the deflection angle definition human-machine interactive information of people's face and with USB mouse mode and compunication; This vision mouse apparatus is suitable for multimedia application and disabled people scholar uses a computer.

Images