CN107545302A - A kind of united direction of visual lines computational methods of human eye right and left eyes image - Google Patents

Abstract

The invention provides a line-of-sight calculation method for the combination of left and right eye images of human eyes, including: an extraction model of binocular information, inputting human eye images, and automatically extracting the left eye and right eye contained in the image respectively through the dual-channel model The information features of the human eyes; the extraction model of the joint information features of the human eye, input the user's binocular image, and the model combines the information of the eyes to extract the joint information features of the human eyes; a joint algorithm is invented, and the three-dimensional line of sight direction is calculated by inputting the feature information. One of the applications of the present invention is virtual reality and human-computer interaction. Its principle is to calculate the direction of the user's line of sight by taking pictures of the user's eyes, so as to interact with the intelligent system interface or virtual reality objects. The invention can also be widely used in the fields of training, game entertainment, video monitoring, medical monitoring and the like.

Description

A gaze direction calculation method based on the combination of human left and right eye images

technical field

The invention relates to the fields of computer vision and image processing, in particular to a method for calculating the line of sight direction of human left and right eye images.

Background technique

Gaze tracking/eye tracking is of great significance for user behavior understanding and efficient human-computer interaction. More than 80% of human perceptible information is received by the human eye, and more than 90% of it is processed by the visual system. Therefore, line of sight is an important clue to reflect the interaction process between people and the outside world. In recent years, due to the rapid development of virtual reality technology and human-computer interaction technology, the application value of gaze tracking technology has gradually become prominent; on the other hand, gaze direction calculation is still a very challenging problem in the field of computer vision.

The current eye-tracking technology is fundamentally divided into two types: appearance-based eye-tracking technology and model-based eye-tracking technology. In the current environment, since model-based eye-tracking technology often has a high accuracy rate, most people focus on the research of model-based eye-tracking methods. The model-based gaze tracking technology requires the experimenter to provide many geometric features, such as the direction of the pupil, and use this to build an eye model, and use the model to predict the direction of people's gaze. Because of this, model-based eye-tracking technology has the following disadvantages: 1) expensive equipment and instruments are needed. Model-based gaze tracking technology predicts the participant's gaze direction by building an eye model or other geometric models, so it needs to use some unique equipment to extract some geometric features of the participant's eyes, and establish Model. 2) Model-based eye-tracking techniques need to be performed in strict indoor environments. Because the geometric features that need to be provided by the experimenter in the model-based eye-tracking technology are generally measured by infrared rays, and some other sources of interference, such as too much infrared light contained in sunlight, will affect the measurement results of the instrument. It causes very serious interference, so the measurement equipment also needs to be placed in a strict indoor environment, so as to avoid the interference of other infrared light such as sunlight. 3) Model-based gaze tracking technology requires high-resolution images for training, so it has a limited working distance, generally no more than 60cm. Therefore, model-based experimental methods cannot be generally used in most common environments.

On the contrary, the appearance-based gaze tracking technology directly learns various information from human eye pictures, and establishes the mapping relationship between the picture and human eye sight, so as to obtain the direction of sight. It does not have the above-mentioned model-based Due to the various limitations of the eye-tracking technology, it only needs to take a picture of the appearance of the human eye through an ordinary camera. This condition makes the appearance-based eye-tracking technology universally applicable, and also makes the appearance-based eye-tracking technology have an unparalleled application prospect. However, since the appearance-based gaze tracking technology does not have strict requirements on sampling tools and sampling environments, the input data of the model is often affected by various environmental factors, such as lighting, participants, and head positions. The intensity of light will make the picture bright or dark. In an extremely dark environment, it is even more difficult for people to distinguish a person's eye image from the picture; similarly, the difference in head position has a great impact on the sampling of human eyes. For the same person, the human eye images obtained by taking its frontal photo and sideways photo are different. The above factors make the input data of the appearance-based eye-tracking technology have a lot of noise information, which is exactly a challenge faced by the appearance-based eye-tracking technology. Because of this defect, the appearance-based eye-tracking The technique, in terms of accuracy, is far inferior to model-based eye-tracking methods. At the same time, the current appearance-based gaze tracking methods often use the user's monocular image as input information. However, in practical applications, each time the user information is collected, it is often the user's binocular image at a certain moment. Inputting binocular images at the same moment separately is also ignoring the correlation of binocular images at a certain moment.

In recent years, a variety of new models have also emerged as the times require. Among these various models, the performance of neural networks is particularly prominent. As a category of neural networks, convolutional neural networks in deep learning are very popular. Since the convolutional neural network in deep learning has the characteristics of local perception, it can extract the local features of the picture well and retain the local relevant information of the picture. At the same time, due to the weight sharing, the convolutional neural network in deep learning The convolutional neural network does not need to consume a lot of time for training. Therefore, the convolutional neural network in deep learning is particularly outstanding in various image processing tasks, such as: image classification, target detection, and semantic segmentation. . At the same time, in recent years, the rapid development of hardware has made the convolutional neural network in deep learning perform better in image processing methods. However, there is no similar literature report on the method for determining the direction of sight of the human eye.

Contents of the invention

The technology of the present invention solves the problem: overcomes the deficiencies of the prior art, and provides a method for calculating the line of sight direction of the left and right eye images of the human eye, by using the neural network to extract the information factors contained in the image, and adjusting the neural The network model finally predicts the direction of sight of both eyes, mainly by combining the image information of both eyes, so as to solve the problem of large noise in the monocular image input in the appearance-based gaze tracking method, thus realizing high-precision 3D gaze direction prediction.

The technical solution of the present invention: a method for calculating the line of sight direction of the combination of left and right eye images of human eyes, comprising the following steps:

(1) Capture the user's face image, locate the left eye or right eye area, preprocess the human eye image, realize the correction of the head position, and obtain the human eye image with a fixed pixel size;

(2) Establish a dual-channel model, input the image information of the left eye and the right eye in the human eye image respectively, use the deep neural network model to extract and output the information features of the left eye and the right eye respectively;

(3) Establish a single-channel model, input the image information of the left eye and the right eye, use the deep neural network model to extract and output the joint information features of the left and right eye images;

(4) Using the regression analysis method, combining the information features of the left and right eyes and the joint information features of the left and right eye images, and through joint optimization, predict the three-dimensional line of sight directions corresponding to the two eyes respectively; or use the information of the left and right eyes alone Feature or left and right eye image joint information features, using the method of regression analysis, after optimization, predict the three-dimensional line of sight direction corresponding to both eyes

Described step (2) establishes two-channel model, input the image information of left eye and right eye in the human eye image respectively, extract respectively through two-channel model and output the concrete process of the information characteristic of left eye and right eye as follows:

(21) Input the fixed-sized left-eye and right-eye images _Il and _Ir after correction into the dual-channel model, and _Il and _Ir are processed by one channel respectively;

(22) Each channel is a deep neural network model, and the model performs convolution, pooling, and full connection operations on the input human eye image, and outputs a fixed-length feature vector;

(23) The fixed-length feature vector generated by each channel is the information feature of the corresponding input image extracted by the deep neural network, and the information features generated by the two channels are connected to obtain the final information of the left and right eyes feature.

The step (3) establishes a single-channel model, inputs the picture information of the left eye and the right eye, and uses the single-channel model to extract and output the joint information feature of the left and right eye images. The specific process is as follows:

(31) Input the corrected fixed-size human eye image into the single-channel model;

(32) Use the deep neural network model to perform convolution, pooling, and full connection operations on the images of the left and right eyes respectively, and output the simplified information features of the left and right eyes;

(33) Connect the information features of the left and right eyes, add multiple fully connected layers after the deep neural network model, use the fully connected layers to merge the information features of the left and right eyes, and finally obtain the joint information features of the left and right eye images.

The step (4) utilizes the method of regression analysis, combines the information characteristics of the left eye and the right eye, and the joint information characteristics of the left and right eye images, and through joint optimization, the specific process of predicting the three-dimensional line of sight directions corresponding to the two eyes is as follows:

(41) Input the corrected left eye and right eye images I _l and I _r, and the real left eye line of sight direction g _l and the real right eye line of sight direction g _r corresponding to the image;

(42) Using the deep neural network model proposed in step (2) and step (3), extract the corresponding binocular information feature and joint information feature of the image;

(43) Connect all the extracted information features or use a single feature as the overall feature, and use the method of regression analysis to obtain the predicted left eye sight direction f(I) _l and the predicted right eye sight direction f(I) _r ;

(44) Use the angle difference as the error value, and use the method of gradient descent to iteratively optimize the model, so that the predicted line of sight direction is getting closer and closer to the real line of sight direction;

(45) Select the model whose predicted line-of-sight direction is closest to the real line-of-sight direction as the final model. The model obtains the predicted line-of-sight direction by inputting the human eye image, and takes this line-of-sight direction as the final prediction result.

Compared with other eye-tracking methods, the beneficial features of the present invention are:

(1) Invented the information feature extraction model of both eyes, which can extract the information features of both eyes, and can also use the feature information of both eyes alone to predict the line of sight direction, and the result is still better than the general monocular line of sight tracking method;

(2) Considering that there is a certain correlation between the binocular images at the same moment, an extraction model of human eye joint information features is invented, which can extract the correlation feature information of both eyes, effectively combining the image information of both eyes, and at the same time Only using the correlation feature information of both eyes to predict the gaze direction, the result is still better than the general monocular gaze tracking method;

(3) A multi-channel neural network is established. The network can more accurately predict the three-dimensional line of sight direction of the two eyes through the method of regression by inputting feature information. The problem with inaccurate results.

Description of drawings

Fig. 1 is a schematic diagram of the network structure of the present invention, wherein a is a dual-channel model in step (2) in the summary of the invention, b is a single-channel model in step (3) in the summary of the invention, and c is a step in the summary of the invention ( 4) line-of-sight prediction model;

Fig. 2 is a schematic diagram of the basic neural network structure of the present invention;

Fig. 3 is the overall structural diagram of the calculation method based on the user's binocular analysis and matching line-of-sight direction of the present invention;

Fig. 4 is a flow chart of model training in the present invention.

detailed description

The specific implementation of the present invention will be described in detail below in conjunction with the accompanying drawings.

The present invention provides a line-of-sight calculation method for the combination of left and right eye images of human eyes, which inputs the information characteristics of human eyes to predict the line-of-sight direction of people's eyes. Information features used in . This method has no additional requirements on the system, and only uses the human eye image captured by a single camera as input. At the same time, by combining the image information of both eyes, the present invention can eliminate some error situations with relatively large monocular noise, thereby achieving better robustness than other similar methods.

First, for the acquisition of human eye images, the present invention includes the following procedures. Using a single camera, capture an image that contains areas of the user's face. Use existing face analysis methods to locate the left or right eye area. The extracted human eye image is preprocessed to obtain a human eye image with a fixed pixel size corrected for the head position.

Secondly, a dual-channel deep neural network model that extracts the information features of the left and right eyes at the same time was invented. On the premise of inputting the binocular images, the left-eye and right-eye images enter a channel respectively, and after being processed separately by each channel, respectively The characteristic information of the left eye and the right eye is obtained.

Furthermore, a single-channel deep neural network model for the extraction of human eye joint information features for binocular image input was invented. On the premise of inputting binocular images, a deep neural network model was established. The model simultaneously processes the image information of the left and right eyes. Combined with the processed image information, the model further processes the combined information to obtain joint information features about the left and right eyes.

Finally, by combining the above two network models, a method for determining the gaze direction of the left and right eye images of the human eye is invented. Methods A multi-channel deep neural network line-of-sight prediction model was established. By fusing the above two information feature extraction models, the obtained information features of both eyes and the joint information features of both eyes were obtained, and the predicted line-of-sight direction of both eyes was obtained through regression analysis. The angular deviation between the predicted line of sight direction and the real line of sight direction is used to measure the quality of the current model. The model uses the gradient descent method for autonomous optimization, by using the formula:

Calculate the angle deviation, where n represents the number of input image pairs, and aim to reduce the angle deviation, and continuously optimize the model. In the process of joint optimization, whenever a pair of human eye images and the real line of sight direction are input, the An iterative optimization of the model is performed. When all known image information is input, the optimization process ends and the final model is obtained. In practical applications, the model directly predicts the gaze direction of the human eye image by receiving a pair of brand new human eye images.

At the same time, the gaze direction estimation method is scalable, and the information features or joint information features of the left and right eyes of the human eye can be used for regression analysis to obtain the predicted gaze direction of both eyes, and the real gaze direction of the input image and the output prediction gaze direction can be used The average angular deviation between them is used as an error term to adjust the prediction model adaptively. Similarly, the gaze direction estimation method is additive. After obtaining the information features of both eyes and the joint information features of both eyes, some other related information features can be directly added, and regression analysis and judgment can be performed based on all the features.

Describe in detail below again, referring to Fig. 1 schematic diagram of the network structure of the present invention, specifically as follows:

(a) in Fig. 1 is a schematic structural diagram TE-I of a dual-channel deep neural network model for simultaneously extracting left-eye and right-eye information features. The model inputs the image of both eyes, and the image is processed by a convolutional neural network (CNN)-based network to output the feature information of the left eye and the right eye respectively. The feature information is called the feature of the eyes, and the line of sight direction of the eyes can also be predicted by the features of the eyes. ;

(b) in Figure 1 is a schematic diagram of the single-channel deep neural network model structure TE-II for simultaneously extracting information features of the left eye and the right eye. After inputting binocular images, the images are firstly processed by a CNN-based network to obtain independent feature information, and then pass through a fully connected layer to fuse the independent feature information to finally obtain binocular correlation information. Similarly, only From the correlation information of both eyes, the line of sight direction of both eyes can also be predicted;

(c) in Figure 1 is a schematic structural diagram of the network model TE-A of the line-of-sight direction calculation method for the combination of left and right eye images of the human eye, by combining the two types of (a) in Figure 1 and (b) in Figure The model structure obtains binocular features and binocular correlation features at one time, and uses this information as an overall feature to obtain the gaze direction of both eyes through regression analysis.

In the above three models, due to the consideration of the difference in the position of the user’s head when the camera is used to capture the appearance of the human eye from the front, the captured human eye images will have different deformations, although the human eye images were initially adjusted accordingly. Transformation to eliminate the influence of the head position, but the influence cannot be completely eliminated, so when predicting and analyzing the direction of sight, the head position vector is also added to the final feature set.

Refer to Fig. 2 for a schematic diagram of the basic neural network structure of the present invention. In order to extract excellent feature information from images, considering the excellent performance of convolutional neural network in image processing, the method uses CNN network as the basic network for feature extraction. The input of the network is a 36×60 grayscale image, and the output is an x-dimensional feature, and the specific value of x can be set by yourself. After the image is input, it first goes through a layer of convolution, the size of the convolution kernel is set to 5×5, and the number of output channels is set to 20. After the first layer of convolution, the output is 20 images of 32× 56-sized pictures, and then, the pictures go through the maximum pooling layer, and after passing through 2×2 pooling, 20 16×28 pictures are output. Then, the 20 pictures are convolved again, the convolution kernel is still 5×5, the output channel is 50, and a total of 50 pictures of 12×24 are output, and then these 50 pictures are subjected to 2×2 maximum pooling Layer, get 50 6×12 pictures. Finally, the 50 6×12 pictures are spread out to obtain 50×6×12 numbers, and the final desired x-dimensional features are obtained through the fully connected layer.

Refer to FIG. 3 for an overall structural diagram of a method for determining the line-of-sight direction combined with images of the left and right eyes of the present invention. The present invention predicts and analyzes the user's binocular three-dimensional line of sight direction by independently constructing a neural network. Methods By inputting a fixed-size human eye grayscale image and a head angle vector, a 1506-dimensional feature vector is obtained, and then a 6-dimensional binocular gaze direction is obtained through regression analysis. The overall structure of the present invention also includes the network of steps (2) and (3) in the summary of the invention. The overall structure of the network in step (2) is shown in Figure 3 with the above two human eye pictures as the input structure, the network inputs the images of the left eye and the right eye respectively, and then uses the CNN network in Figure 2 to convolve the images, and The final number of features x is set to 1000, and a feature vector with a length of 1000 is obtained; the feature vectors pass through a fully connected layer (FC) respectively, and a 500-dimensional feature vector is obtained respectively; finally, simply combine these two 500 Dimensional feature vectors are concatenated as the output features of the first part. The overall structure of the network in step (3) is shown in Figure 3 with the following two images as input structures. The second part also uses the left-eye and right-eye images as input, and the image is convoluted by using the CNN network, and the final number of features Set x to 500 to obtain 500-dimensional feature vectors, and then simply connect the 500-dimensional feature vectors into a 1000-dimensional vector, and fuse the feature vectors through a fully connected layer to obtain 500-dimensional features vector, and use the 500-dimensional feature vector as the output of the second part; similarly, due to the consideration of the difference in head position, some irreversible effects will be caused on the image, the present invention uses the head position vector as the third Part of the input, without processing, is directly added to the final feature vector. The three parts output 1000-dimensional, 500-dimensional, and 6-dimensional feature vectors respectively, and these vectors are connected to obtain the final feature of 1506 dimensions.

Referring to FIG. 4 , the flow chart of the gaze direction prediction based on the user's binocular image of the present invention, combined with the related specific technologies described above, the specific implementation process of the gaze direction prediction based on the user's binocular image is introduced below.

First, the predictive model is initialized by using a simple method of randomly assigning initial values to the model presented in Fig. 3. Then, the processed human eye image is input. Whenever a pair of human eye images I _l and I _r are input, a pair of predicted three-dimensional line-of-sight directions f(I) _l and f(I) _r can be obtained through the network, respectively Represents the gaze direction of the left and right eyes. Then, by comparing with the original three-dimensional line-of-sight directions g _l and g _r , the predicted angle deviation is obtained, and then the network is continuously optimized by using the gradient descent method to reduce the angle deviation. Whenever a pair of images is input, An iterative adjustment of the network parameters is performed. When all the images are input, the final prediction model is obtained. In the final prediction model, by inputting an image, the line-of-sight direction corresponding to the image can be predicted.

The above is only a representative embodiment of the present invention, and any equivalent transformation made according to the technical solution of the present invention shall belong to the protection scope of the present invention.

Claims (4)

1. A line-of-sight calculation method for the combination of left and right eye images of human eyes, characterized in that, comprising the following steps: (1) Capture the user's face image, locate the left eye or right eye area, preprocess the human eye image, realize the correction of the head position, and obtain the human eye image with a fixed pixel size; (2) Establish a dual-channel model, input the image information of the left eye and the right eye in the human eye image respectively, use the deep neural network model to extract and output the information features of the left eye and the right eye respectively; (3) Establish a single-channel model, input the image information of the left eye and the right eye, use the deep neural network model to extract and output the joint information features of the left and right eye images; (4) Using the regression analysis method, combining the information features of the left and right eyes and the joint information features of the left and right eye images, and through joint optimization, predict the three-dimensional line of sight directions corresponding to the two eyes respectively; or use the information of the left and right eyes alone Features or the joint information features of the left and right eye images, using the method of regression analysis, after optimization, predict the three-dimensional line of sight directions corresponding to the two eyes respectively. 2. the line-of-sight calculation method of the combination of left and right eye images of human eyes according to claim 1, characterized in that: said step (2) sets up a two-channel model, and inputs the image information of left eye and right eye in the human eye image respectively , the specific process of extracting and outputting the information features of the left eye and the right eye through the dual-channel model is as follows: (21) Input the fixed-sized left-eye and right-eye images _Il and _Ir after correction into the dual-channel model, and _Il and _Ir are processed by one channel respectively; (22) Each channel is a deep neural network model, and the model performs convolution, pooling, and full connection operations on the input human eye image, and outputs a fixed-length feature vector; (23) The fixed-length feature vector generated by each channel is the information feature of the corresponding input image extracted by the deep neural network, and the information features generated by the two channels are connected to obtain the final information of the left and right eyes feature. 3. the line-of-sight calculation method of the combination of left and right eye images of human eyes according to claim 1, characterized in that: the step (3) sets up a single-channel model, inputs the picture information of the left eye and the right eye, and uses the single-channel model The specific process of extracting and outputting the joint information features of left and right eye images is as follows: (31) Input the corrected fixed-size human eye image into the single-channel model; (32) Use the deep neural network model to perform convolution, pooling, and full connection operations on the images of the left and right eyes respectively, and output the simplified information features of the left and right eyes; (33) Connect the information features of the left and right eyes, add multiple fully connected layers after the deep neural network model, use the fully connected layers to merge the information features of the left and right eyes, and finally obtain the joint information features of the left and right eye images. 4. The line-of-sight calculation method of the combination of left and right eye images of human eyes according to claim 1, characterized in that: said step (4) utilizes the method of regression analysis, combines the information characteristics of left eye and right eye, and left and right eyes The specific process of predicting the three-dimensional line-of-sight direction corresponding to both eyes is as follows: (41) Input the corrected left eye and right eye images I _l and I _r , and the real left eye sight direction g _l and the real right eye sight direction g _r corresponding to the images; (42) Using the deep neural network model proposed in step (2) and step (3), extract the corresponding binocular information feature and joint information feature of the image; (43) Connect all the extracted information features or use a single feature as the overall feature, and use the method of regression analysis to obtain the predicted left eye sight direction f(I) _l and the predicted right eye sight direction f(I) _r ; (44) Use the angle difference as the error value, and use the method of gradient descent to iteratively optimize the model, so that the predicted line of sight direction is getting closer and closer to the real line of sight direction; (45) Select the model whose predicted line-of-sight direction is closest to the real line-of-sight direction as the final model. The model obtains the predicted line-of-sight direction by inputting the human eye image, and takes this line-of-sight direction as the final prediction result.