CN102043952A - Eye-gaze tracking method based on double light sources - Google Patents

Abstract

The invention discloses a line-of-sight tracking method based on dual light sources. The specific steps are as follows: (1) Image preprocessing: using the principle of the gray scale difference between the pupil and the reflection point, the collected face image is pre-processed to extract the human face image. Two reflection points and the pupil area in the eye image, and calculate the coordinates of the two reflection points and the pupil center in the image coordinate system; (2) Gaze point estimation: According to the triangle formed by the pupil center and the two reflection points in the image and the screen The triangle formed by the upper gaze point and the two infrared light sources is approximated as a pair of similar triangles to determine the approximate position of the gaze point on the screen; (3) gaze point correction to obtain the final gaze point position. The invention does not need to measure the distance between people and the screen, has low computational complexity, is robust to head movement, is natural and comfortable in the experiment process, is easy to implement, and the estimation error is within the tolerance range of the eye-tracking system application error.

Description

A kind of sight tracing based on two light sources

Technical field

The present invention relates to a kind of sight tracing, belong to video, multimedia signal processing technique field based on two light sources.

Background technology

Along with the develop rapidly of computer science and technology and industry, domestic consumer's sharp increase of computing machine, intelligentized man-machine interaction also receives publicity day by day.As the important channel that the mankind obtain external information, sight line also can be brought into play its great potential in intelligent man-machine interaction, becomes the effective tool of intelligent interaction.For example, we can express our subjective thought and then realize that things ground is directly controlled to external world by sight line

T.Hutchinson etc. took the lead in having proposed in 1989 carrying out mutual thought with sight line and computing machine.Its tempting application prospect has attracted countless domestic and international research persons from each side such as image pre-service, mapping algorithm, corrections eye tracking to be studied.Nowadays, the eye tracking technology is increasingly mature, it uses the approval that also constantly obtains people,, development of games auxiliary, the appearance design 3D free viewpoint video in epoch in vogue of testing and assessing from the disabled person, none brings convenience for our life, has effectively promoted intelligent science and technology development especially.

The method of eye tracking has electroculogram (EOG), iris-sclera edge, corneal reflection, pupil-corneal reflection vector etc. multiple, and wherein, because its precision is higher, user's comfort is high and enjoy favor based on the method for pupil-corneal reflection vector.Utilizations such as D.H.Yoo four light sources in calendar year 2001 propose can comparatively accurately realize the estimation of sight line based on the gaze tracking system of the constant algorithm of double ratio.Yet, because be subjected to that hardware is integrated, the influence of execution speed and application prospect, people more are partial to adopt still less light source to realize the estimation of sight line.At present, want to realize that comparatively accurate sight line estimation needs two infrared light supplies at least under the situation freely at head movement.And existing two light source sight tracings calculate more complicated, even if allow head moving, correct moving scope also has very big restriction.Therefore find a kind of simple and practical eye tracking mapping method to have important theoretical research meaning and great actual application value.

Summary of the invention

At existing two light source sight tracing calculation of complex, to the restricted big problem of head range of movement, the invention provides a kind of computation complexity low, to little, the practical sight tracing of head limit movement based on two light sources.

According to the image-forming principle of watching characteristic and camera attentively of human eye, as shown in Figure 4, two light source L on the screen ₁, L ₂The reflection spot V that on eye cornea form corresponding with it ₀₁, V ₀₂Line and blinkpunkt Q on the screen and the P of pupil center ₀Line meet at the corneal curvature center C of human eye; And, according to the refraction at spherical surface principle, the reflection spot V of two infrared light supplies on cornea ₀₁, V ₀₂And the P of pupil center ₀With they picture V after the cornea refraction at spherical surface ₁, V ₂And therefore P in same footpath upwards, uses V ₁, V ₂And P replaces V respectively ₀₁, V ₀₂And P ₀Can not influence the blinkpunkt results estimated.

Set up in the process in mapping, because people's eyeball is a spheroidite, so the reflection spot V that two infrared light supplies form in human eye ₁, V ₂Though and pupil center is inequality, of slight difference at the coordinate figure of the refraction point on the cornea on Z-direction.Calculate by analysis: these 3 coordinate figures on the Z axle can think approximate identical, the blinkpunkt result is estimated that the error that is caused is 0.9887cm to the maximum on X-axis, on Y-axis, be 0.7852cm to the maximum, and the trend of error presents compartmentalization and distributes, therefore can calibrate in conjunction with secondary and reduce because the approximate error that produces, the error of the feasible blinkpunkt that finally estimates is within the tolerance interval in actual applications.

Sight tracing based on two light sources of the present invention, concrete steps are as follows:

(1) image pre-service: the principle of utilizing pupil and reflection spot gray scale difference, the facial image that collects is carried out pre-service, extract in the eye image two reflection spots and pupil region, and calculate two reflection spots and the pupil center coordinate in image coordinate system, wherein image coordinate system is a true origin with the upper left corner of image, horizontal direction is an X-axis, and vertical direction is a Y-axis;

(2) blinkpunkt is estimated: the triangle of blinkpunkt and two infrared light supply formations is approximately the apparent position that a pair of similar triangles are determined blinkpunkt on the screen on triangle that constitutes according to pupil center in the image and two reflection spots and the screen.

(3) blinkpunkt is proofreaied and correct: the method for utilization secondary calibration is proofreaied and correct the intrinsic deviation of human eye's visual axis and optical axis and because of being similar to the evaluated error that produces, is obtained final blinkpunkt position.

The specific implementation step that described step (2) blinkpunkt is estimated is:

A. draw blinkpunkt and two leg-of-mutton approximate similarity sexual intercourse that infrared light supply constitutes on triangle that pupil center and two reflection spots constitute and the screen according to the image-forming principle of watching characteristic and camera attentively of human eye;

B. estimate the apparent position of blinkpunkt on the screen according to the triangle similarity.

The specific implementation step that described step (3) blinkpunkt is proofreaied and correct is:

The a-quadrant is divided: number and distribution according to scaling point are five zones with whole screen divider;

1 calibration of B: the scaling point by the screen centre position carries out some calibration;

C zone location: according to the position of some calibration back blinkpunkt and the zone at the definite blinkpunkt place of relation, the position between each scaling point;

D secondary calibration: according to the zone at blinkpunkt place, the blinkpunkt after once calibrating is carried out the secondary calibration, obtain final blinkpunkt position with the scaling point in the blinkpunkt region.

The triangle that blinkpunkt and two infrared light supplies constitute on triangle that the present invention constitutes according to the pupil center in the image and two reflection spots and the screen is approximately the apparent position that a pair of similar triangles are determined blinkpunkt on the screen, calibrate by secondary again and proofread and correct inherent variability and the approximate evaluated error that is caused of algorithm between human eye's visual axis and the optical axis, obtain comparatively accurate blinkpunkt estimated value.This method need not to measure the distance of head distance screen, and computation complexity is low, and to the head movement strong robustness, experimentation is comfortable naturally, is easy to realize, is convenient to integrated actual product.

Description of drawings

Fig. 1 is the system hardware figure that realizes the inventive method.

Fig. 2 is the process flow diagram of the inventive method.

Fig. 3 is the synoptic diagram of image preprocessing process in the inventive method.

Fig. 4 is video camera imaging and people's eye fixation principle schematic.

Fig. 5 is the approximate substitution synoptic diagram of coordinate position in the inventive method.

Fig. 6 is the leg-of-mutton synoptic diagram of two approximate similarities.

Fig. 7 is that scaling point distributes and screen area is divided synoptic diagram

Fig. 8 is a blinkpunkt estimated result synoptic diagram.

Fig. 9 is that the blinkpunkt estimated accuracy is analyzed synoptic diagram.

Embodiment

Realization the present invention is based on two light sources sight tracing hardware system as shown in Figure 1, comprise a gray scale video camera, two infrared light supplies and a personal computer.Personal computer adopts the 2.60GHZ Pentium Dual Core, computer screen size 34 * 27cm (wide * height), it is 694 * 1040 gray scale video camera that a resolution is adorned in computing machine below, it is motionless that camera position keeps in experimentation, and one 1 watt infrared light supply L is equipped with in the lower left corner of display and the lower right corner respectively ₁And L ₂The tester is sitting in apart from the position of screen 60--70cm, and head can move in the scope of 20 * 20 * 10cm (wide * length * degree of depth).

Fig. 2 has provided the process flow diagram of the sight tracing based on two light sources of the present invention, and according to this flow process, concrete implementation step is as follows:

1. determine reflection spot center and the pupil center coordinate in image coordinate system in the image by the image pre-service, its process as shown in Figure 3.According to the gray scale difference of pupil and corneal reflection point, extract pupil and corneal reflection point zone, and obtain their the coordinate position p of center in image coordinate system respectively _i(p _x, p _y), v _1i(v _1ix, v _1iy), v _2i(v _2ix, v _2iy).Wherein image coordinate system is a true origin with the upper left corner of image, and horizontal direction is an X-axis, and vertical direction is a Y-axis.

2. blinkpunkt is estimated.The triangle that blinkpunkt and two infrared light supplies constitute on triangle that constitutes according to the pupil center in the image and two reflection spots and the screen is approximately a pair of similar triangles and determines blinkpunkt apparent position on the screen.Its specific implementation step is:

(1), draws blinkpunkt and two leg-of-mutton approximate similarity sexual intercourse that infrared light supply constitutes on triangle that pupil center and two reflection spots constitute and the screen according to the image-forming principle of watching characteristic and camera attentively of human eye shown in Figure 4.Its concrete steps are as follows:

(a) according to the image-forming principle of camera, with corneal reflection point in the facial image and pupil center the coordinate conversion in camera coordinates system in world coordinate system, its transformational relation as shown in the formula:

p _i(p _ix，p _iy)→p(p _x，p _y，p _z)＝p(pixelpitch _c(p _ix-c _certer)+O _x，pixelpitch _r(p _iy-r _center)+O _y，-λ+O _z)

v _1i(v _1ix，v _1iy)→v ₁(v _1x，v _1y，v _1z)＝v ₁(pixelpitch _c(v _1ix-c _certer)+O _x，pixelpitch _r(v _1iy-r _center)+O _y，-λ+O _z)

v _2i(v _2ix，v _2iy)→v ₂(v _2x，v _2y，v _2z)＝v ₂(pixelpitch _c(v _2ix-c _center)+O _x，pixelpitch _r(v _2iy-r _center)+O _y，-λ+O _z)

Wherein, space coordinates are initial point with the lower left corner of television screen, are the x axle with the axle of the horizontal direction that is parallel to television screen, are y with the axle of the vertical direction that is parallel to television screen, are the z axle with the axle perpendicular to the direction of television screen directed towards user.Pixe; Pitch _c, pixelpitch _r, c _Center, r _CenterAll are intrinsic parameters of camera, O _x, O _y, O _zBe the coordinate figure of camera light node on a world coordinate system coordinate axis.λ is the constant relevant with camera focus, focal length and object distance.From above-mentioned conversion relational expression as can be seen, by a p (p _x, p _y, p _z), v ₁(v _1x, v _1y, v _1z) and v ₂(v _2x, v _2y, v _2z) plane parallel that constitutes is in screen and Δ pv ₁v ₂Similar in appearance to Δ p _iv _1iv _2i

(b) according to the image-forming principle of camera, the P (P of pupil center _x, P _y, P _z) on the line of pO, the reflection spot V of infrared light supply on cornea ₁(V _1x, V _1y, V _1z), V ₂(V _2x, V _2y, V _2z) respectively at v ₁O and v ₂On the line of O.Strictly speaking, P _z, V _2zAnd V _2zBe unequal, but they are more or less the same, so let us can think P _z=V _2z=V _1z, promptly put P and V ₂Use straight line OP and plane Z=V respectively _2zIntersection point P " and straight line OV ₂With plane Z=V _2zIntersection point V ₂" replace, as shown in Figure 6, and Δ P " V ₁V ₂" similar in appearance to triangle Δ pv ₁v ₂If human eye center of curvature C and P " line and the intersection point of screen be Q, then Q is approximate blinkpunkt estimated value, and Δ QL ₁L ₂Similar in appearance to Δ P " V ₁V ₂", so Δ QL ₁L ₂Similar in appearance to Δ pv ₁v ₂And similar in appearance to Δ p _iv _1iv _2i

(2) be respectively L through measuring the coordinate that can obtain two infrared light supplies ₁(0,0) and L ₂(0,34), L _1xAnd L _2xBe positioned at same horizontal line.By Δ QL ₁L ₂Similar in appearance to Δ p _iv _1iv _2i, as shown in Figure 5, utilize the character of similar triangle, can obtain blinkpunkt coordinate Q (Q approximate on the screen according to following equation _x, Q _y).

$\left\{\begin{array}{c}\frac{Q_x}{L_{2x}-L_{1x}}=\frac{{\mathrm{<figure-callout\; id="1"\; label="v"\; filenames="HDA0000042266260000043.png"\; state="\{\{state\}\}">v</figure-callout>}}_{1\mathrm{ix}}-{\mathrm{<figure-callout\; id="1"\; label="p\; ix\; v"\; filenames="HDA0000042266260000043.png"\; state="\{\{state\}\}">p</figure-callout>}}_{\mathrm{ix}}}{v_{}}\\ \frac{Q_y}{L_{2x}-L_{1x}}=\frac{{\mathrm{<figure-callout\; id="1"\; label="v"\; filenames="HDA0000042266260000043.png"\; state="\{\{state\}\}">v</figure-callout>}}_{1\mathrm{iy}}-p_{\mathrm{iy}}}{v_{1\mathrm{ix}}-{\mathrm{<figure-callout\; id="2"\; label="v"\; filenames="HDA0000042266260000012.png"\; state="\{\{state\}\}">v</figure-callout>}}_{2\mathrm{ix}}}\end{array}\right.\&DoubleRightArrow;\left\{\begin{array}{c}{Q}_x=\frac{v_{1\mathrm{ix}}-p_{\mathrm{ix}}}{v_{1\mathrm{ix}}-{\mathrm{<figure-callout\; id="2"\; label="v"\; filenames="HDA0000042266260000012.png"\; state="\{\{state\}\}">v</figure-callout>}}_{2\mathrm{ix}}}*(L_{2x}-L_{1x})\\ Q_y=\frac{v_{1\mathrm{iy}}-p_{\mathrm{iy}}}{v_{1\mathrm{ix}}-{\mathrm{<figure-callout\; id="2"\; label="v"\; filenames="HDA0000042266260000012.png"\; state="\{\{state\}\}">v</figure-callout>}}_{2\mathrm{ix}}}*(L_{2x}-L_{1x})\end{array}\right.$

L wherein _1xAnd L _2xBe respectively infrared light supply L ₁And L ₂Coordinate figure on X-axis.

3,,, the position of the blinkpunkt that is calculated in the step 2 watches the position attentively so need being only accurately human eye through the calibration correction because vision optical axis and the intrinsic deviation of the optical axis.Adopt the secondary calibration mode, also can reduce the error that produces owing to the approximate substitution in the algorithm when proofreading and correct the optical axis and optical axis inherent variability, concrete steps are as follows:

(1) area dividing.Number and distribution according to scaling point in the system are A with whole screen divider ₀, A ₁, A ₂, A ₃And A ₄Five zones, as shown in Figure 7.

(2) some calibration.Scaling point by the screen centre position carries out some calibration.Its concrete grammar is as follows:

If the true coordinate value of scaling point a is (x_real_a, y_real_a), and utilize coordinate figure that algorithm estimates for (x_gaze_a, y_gaze_a), then scaling point a is at the poor x_error_a between actual value and the estimated value on X-axis and the Y-axis, and y_error_a can be represented by the formula:

x_error_a＝x_gaze_a-x_real_a

y_error_a＝y_gaze_a-y_real_a

Suppose the blinkpunkt Q that the algorithm with this pair light source estimates _i(i=1,2,3, L) with all the other four scaling point b, c, d, the coordinate position of e is respectively (x_gaze_Q _i, y_gaze_Q _i) (i=1,2,3, L), (x_gaze_b, y_gaze_b), (x_gaze_c, y_gaze_c), (x_gaze_d, y_gaze_d) and (x_gaze_e, y_gaze_e), then through the coordinate (x_gazel_Q of center scaling point a after once calibrating _i, y_gazel_Q _i), (x_gazel_b, y_gazel_b), (x_gazel_c, y_gazel_c), (x_gazel_d, y_gazel_d), (x_gazel_e y_gazel_e) can be expressed as:

x_gazel_Q _i＝x_gaze_Q _i+w _ix*x_error_a，y_gazel_Q _i＝y_gaze_Q _i+w _iy*y_error_a

x_gazel_b＝x_gaze_b+w _bx*x_error_a，y_gazel_b＝y_gaze_b+w _by*y_error_a

x_gazel_c＝x_gaze_c+w _cx*x_error_a，y_gazel_c＝y_gaze_c+w _cy*y_error_a

x_gazel_d＝x_gaze_d+w _dx*x_error_a，y_gazel_d＝y_gaze_d+w _dy*y_error_a

x_gazel_e＝x_gaze_e+w _ex*x_error_a，y_gazel_e＝y_gaze_e+w _ey*y_error_a

Wherein, weight coefficient w _Ix, w _Bx, w _Cx, w _Dx, w _Ex, w _Iy, w _By, w _Cy, w _Dy, w _EyThe distance dependent on X-axis or Y-axis with corresponding point and centre coordinate point.

(3) zone location.According to the position of some calibration back blinkpunkt and the zone at the definite blinkpunkt place of relation, the position between each scaling point.

(4) secondary calibration.According to the zone at blinkpunkt place, with the scaling point in the blinkpunkt region blinkpunkt after once calibrating is carried out the secondary calibration, obtain final blinkpunkt position, its coordinate position can be expressed as:

X_gaze2_Q _i=x_gazel_Q _i, y_gaze2_Q _i=y_gazel_Q _i(at regional A ₀)

X_gaze2_Q _i=x_gazel_Q _i+ x_error_b, y_gaze2_Q _i=y_gazel_Q _i+ y_error_b is (at regional A ₁)

X_gaze2_Q _i=x_gazel_Q _i+ x_error_c, y_gaze2_Q _i=y_gazel_Q _i+ y_error_c is (at regional A ₂)

X_gaze2_Q _i=x_gazel_Q _i+ x_error_d, y_gaze2_Q _i=y_gazel_Q _i+ y_error_d is (at regional A ₃)

X_gaze2_Q _i=x_gazel_Q _i+ x_error_e, y_gaze2_Q _i=y_gazel_Q _i+ y_error_e is (in zone 4 ₄)

Should be based on two light source sight tracings of space similar triangles and secondary calibration, need not to measure the distance between head and the screen, computing method are simple, though exist approximate error can not surpass their maximal value 0.9887cm and 0.7852cm in algorithm on X-axis and Y-axis.And because the custom characteristic of people's eye fixation, the error of its blinkpunkt estimated result presents zonal distribution, as shown in Figure 8.Fig. 9 has provided blinkpunkt estimated accuracy analysis chart.The evaluated error result of the blinkpunkt that is used to test is as shown in the table:

Blinkpunkt	X_error(cm)	Yerror(cm)
			1	0.5050	0.3143
2	0.5132	0.8293
			3	0.5299	0.3748
4	0.6994	0.0992
			5	0.9113	0.2203
6	0.5349	0.3853
			7	0.2769	0.0475
8	1.0562	0.1781
			9	0.7784	0.4744
10	0.3104	0.6536
			11	0.2331	0.3049
12	1.1230	0.1768

13	1.0073	0.2070
			14	0.7716	0.5990
15	0.9398	0.6581
			16	0.9398	0.5310
Average	0.6956	0.3784

Its average error on X-axis and Y-axis is respectively 0.6956cm and 0.3784cm, so method of the present invention can be used for actual application fully.

Claims (3)

1. A line-of-sight tracking method based on dual light sources, characterized in that, comprising the following steps: (1) Image preprocessing: Using the principle of grayscale difference between the pupil and the reflection point, the collected face image is preprocessed, two reflection points and the pupil area in the human eye image are extracted, and two reflection points are calculated And the coordinates of the pupil center in the image coordinate system, wherein the image coordinate system takes the upper left corner of the image as the coordinate origin, the horizontal direction is the X axis, and the vertical direction is the Y axis; (2) Gaze point estimation: determine the approximate position of the gaze point on the screen according to the triangle formed by the center of the pupil and the two reflection points in the image and the triangle formed by the gaze point and the two infrared light sources on the screen as a pair of similar triangles; (3) Fixation point correction: use the method of secondary calibration to correct the inherent deviation between the visual axis and the optical axis of the human eye and the estimation error caused by approximation, and obtain the final fixation point position. 2. the line-of-sight tracking method based on dual light sources according to claim 1, is characterized in that, the concrete realization step of described step (2) gaze point estimation is: A. According to the gaze characteristics of the human eye and the imaging principle of the camera, the approximate similarity relationship between the triangle formed by the center of the pupil and two reflection points and the triangle formed by the fixation point on the screen and two infrared light sources is obtained; B. Estimate the approximate position of the gaze point on the screen from the triangle similarity. 3. the line-of-sight tracking method based on dual light sources according to claim 1, is characterized in that, the specific implementation steps of described step (3) fixation point correction are: A area division: divide the entire screen into five areas according to the number and distribution of calibration points; B One-point calibration: perform one-point calibration through the calibration point in the middle of the screen; C area positioning: determine the area where the gaze point is located according to the position of the gaze point after one point calibration and the positional relationship between each calibration point; D secondary calibration: According to the area where the gaze point is located, use the calibration points in the area where the gaze point is located to perform secondary calibration on the gaze point after primary calibration to obtain the final gaze point position. the

Images