JP2022124778A - Electronic apparatus - Google Patents

Abstract

To provide an electronic apparatus capable of accurate visual line detection independent of a distance to an eye ball.SOLUTION: The electronic apparatus, which is capable of visual line detection to detect a visual line of a user, comprises: a first light source which applies light of a first wavelength to an eye of the user; a second light source which applies light of a second wavelength different from the first wavelength to the eye of the user; a sensor which receives light for the visual line detection; and control means which switches a light source to use for the visual line detection between the first light source and the second light source.SELECTED DRAWING: Figure 8

Description

The present invention relates to an electronic device having a line-of-sight detection function.

Cameras (including video cameras) that can detect a user's line of sight (line of sight direction) with a line of sight detection function and can select a range-finding point based on the result of the line of sight detection have been put to practical use. For example, in Patent Document 1, when a subject is irradiated with light of a first wavelength and light of a second wavelength to specify an eye region in a skin region, and irradiated with light of the first wavelength, A technique for detecting a line of sight based on an eye image (image of an eye region) is disclosed.

JP 2013-62560 A

However, in the conventional technology disclosed in Patent Document 1, the line of sight is detected by irradiating light of a single wavelength. resulting in decreased accuracy of line-of-sight detection.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an electronic device capable of highly accurate line-of-sight detection regardless of eyeball distance.

In order to achieve the above object, an electronic device of the present invention is an electronic device capable of detecting a line of sight of a user, comprising: a first light source for irradiating light of a first wavelength to the eye of the user; a second light source for irradiating the eyes of the user with light having a second wavelength different from the first wavelength; a sensor for receiving light for detecting the line of sight; and control means for switching between the first light source and the second light source.

According to the present invention, it is possible to provide an electronic device capable of highly accurate line-of-sight detection regardless of eyeball distance.

1 is an external view of a camera; FIG. 1 is a block diagram of a camera; FIG. It is a sectional view of a camera. It is a figure which shows the EVF part of a camera. FIG. 4 is a diagram showing optical paths from infrared LEDs; It is a figure for demonstrating the principle of a line-of-sight detection method. FIG. 4 is a diagram showing an eye image; It is a flow chart of sight line detection operation. FIG. 4 is a diagram for explaining the relationship between the wavelength of an infrared LED and the imaging state of an eye image; It is a flow chart of sight line detection operation.

[First Embodiment]
A first embodiment of the present invention will be described below.

<Description of configuration>
1(a) and 1(b) show the appearance of a camera 1 (digital still camera; interchangeable lens camera), which is an imaging device according to the first embodiment. Note that the present invention is applicable to any electronic device capable of executing line-of-sight detection for detecting the user's line of sight. A user, for example, visually recognizes information such as images and characters displayed on a display device, or visually recognizes an optical image through an eyepiece optical system. These electronic devices may include, for example, mobile phones, game machines, tablet terminals, personal computers, watch-type or spectacle-type information terminals, head-mounted displays, and binoculars.

FIG. 1(a) is a front perspective view, and FIG. 1(b) is a rear perspective view. As shown in FIG. 1(a), the camera 1 has a photographing lens unit 1A and a camera housing 1B. The camera housing 1B is provided with a release button 34, which is an operation member for receiving an imaging operation from a user (photographer). As shown in FIG. 1B, an eyepiece window frame 121 for the user to look into a second display panel 6 (described later) included in the camera housing 1B is arranged on the rear surface of the camera housing 1B. ing. The eyepiece window frame 121 forms the peephole 12 and protrudes outward (to the rear side) with respect to the camera housing 1B. Operation members 41 to 43 for receiving various operations from the user are also arranged on the rear surface of the camera housing 1B. For example, the operation member 41 is a touch panel that accepts touch operations, the operation member 42 is an operation lever that can be pushed down in each direction, and the operation member 43 is a four-way key that can be pushed in each of four directions. The operation member 41 (touch panel) includes a display panel such as a liquid crystal panel, and has a function of displaying an image on the display panel.

FIG. 2 is a block diagram showing the configuration inside the camera 1. As shown in FIG.

The image pickup device 2 is an image pickup device such as a CCD or CMOS sensor, and photoelectrically converts an optical image formed on the image pickup surface of the image pickup device 2 by the optical system of the photographing lens unit 1A, and converts the obtained analog image signal. Output to an A/D converter (not shown). The A/D converter A/D-converts the analog image signal obtained by the imaging element 2 and outputs it as image data.

The photographing lens unit 1A is composed of an optical system including a zoom lens, a focus lens, an aperture, etc., and is attached to the camera housing 1B. is imaged on the imaging surface of The aperture control unit 118, the focus adjustment unit 119, and the zoom control unit 120 each receive an instruction signal from the CPU 3 via the mount contact 117, and drive and control the aperture, focus lens, and zoom lens according to the instruction signal.

The CPU 3 provided in the camera housing 1B reads a control program for each block provided in the camera housing 1B from the ROM 4a of the memory section 4, develops it in the RAM 4b of the memory section 4, and executes it. Thereby, the CPU 3 controls the operation of each block provided in the camera housing 1B. A line-of-sight detection unit 201, a photometry unit 202, an automatic focus detection unit 203, a signal input unit 204, a display device drive unit 208, a light source drive unit 205, and the like are connected to the CPU3. The CPU 3 also transmits signals via the mount contact 117 to the aperture control section 118, the focus control section 119 and the zoom control section 120 arranged in the photographing lens unit 1A. In the first embodiment, the memory unit 4 has a function of storing imaging signals from the imaging element 2 and the line-of-sight detection sensor 30 .

The line-of-sight detection unit 201 A/D-converts the output of the line-of-sight detection sensor 30 when the eyeball image is formed in the vicinity of the line-of-sight detection sensor 30 (the eye image of the eye), and transmits the result to the CPU 3 . The CPU 3 extracts feature points necessary for line-of-sight detection from the eye image according to a predetermined algorithm, which will be described later, and calculates the user's line of sight (line-of-sight position in the image for visual recognition) from the positions of the feature points.

A photometry unit 202 performs amplification, logarithmic compression, A/D conversion, and the like of a signal obtained from the image sensor 2 that also serves as a photometry sensor, specifically, a luminance signal corresponding to the brightness of the object scene. The result is sent to the CPU 3 as field luminance information.

The autofocus detection unit 203 A/D-converts signal voltages from a plurality of detection elements (a plurality of pixels) used for phase difference detection, which are included in the imaging element 2 (for example, CCD), send to The CPU 3 calculates the distance to the object corresponding to each focus detection point from the signals of the plurality of detection elements. This is a well-known technique known as imaging plane phase difference AF. In the first embodiment, as an example, it is assumed that the viewfinder image (visual recognition image) is divided and each of the 180 divided locations on the imaging plane has a focus detection point.

A light source drive unit 205 drives infrared LEDs 18, 19, 22 to 27, which will be described later, based on a signal (instruction) from the CPU 3. FIG.

The image processing unit 206 performs various image processing on the image data stored in the RAM 4b. For example, various image processing for developing, displaying, and recording digital image data, such as correction processing for pixel defects caused by optical systems and image sensors, demosaicing processing, white balance correction processing, color interpolation processing, and gamma processing. done.

A switch SW1 and a switch SW2 are connected to the signal input unit 204 . The switch SW1 is a switch for starting photometry, distance measurement, line-of-sight detection, etc. of the camera 1, and is turned ON by the first stroke of the release button 34 (for example, half-press). The switch SW2 is a switch for starting the photographing operation, and is turned ON by the second stroke of the release button 34 (for example, full depression). ON signals from the switches SW1 and SW2 are input to the signal input unit 204 and transmitted to the CPU3. The signal input unit 204 also receives operation input from the operation member 41 (touch panel), the operation member 42 (operation lever), and the operation member 43 (four-way key) shown in FIG. 1B.

A recording/output unit 207 records data including image data on a recording medium such as a removable memory card, or outputs the data to an external device via an external interface.

The display device driving section 208 drives the display device 209 based on the signal from the CPU3. The display devices 209 are a first display panel 5 and a second display panel 6, which will be described later.

FIG. 3 is a cross-sectional view of the camera 1 cut along the YZ plane formed by the Y-axis and Z-axis shown in FIG.

The shutter 32 and the image sensor 2 are arranged in order along the optical axis of the photographing lens unit 1A.

A first display panel 5 is provided on the rear surface of the camera housing 1B. The first display panel 5 displays menus and images for operating the camera 1 and for viewing and editing images obtained by the camera 1. conduct. The first display panel 5 is composed of a liquid crystal panel with a backlight, an organic EL panel, or the like.

The EVF provided in the camera housing 1B can display menus and images like the first display panel 5 as a normal EVF. It has a configuration that can be reflected in the control of

The second display panel 6 provides the same display as the first display panel 5 (menu display and image display for operating the camera 1 and viewing/editing images obtained by the camera 1) when the user is looking through the viewfinder. display). The second display panel 6 is composed of a liquid crystal panel with a backlight, an organic EL panel, or the like. The size of the second display panel 6 in the X-axis direction (horizontal direction), such as 3:2, 4:3, or 16:9, is larger than the size in the Y-axis direction (vertical direction), as with images captured by a general camera. Consists of long rectangles.

A panel holder 7 is a panel holder for holding the second display panel 6 , and the second display panel 6 and the panel holder 7 are adhered and fixed to form a display panel unit 8 .

The first optical path splitting prism 9 and the second optical path splitting prism 10 are adhered together to form an optical path splitting prism unit 11 (optical path splitting member). The optical path splitting prism unit 11 guides the light from the second display panel 6 to an eyepiece window 17 provided in the peephole 12, and conversely detects the reflected light from the eye (pupil) led from the eyepiece window 17 as a line of sight detection sensor. lead to 30.

The display panel unit 8 and the optical path splitting prism unit 11 are fixed with a mask 33 interposed therebetween and are integrally formed.

The eyepiece optical system 16 is an optical system provided in a portion (eyepiece portion) where the user eyepieces, and is composed of the G1 lens 13 , the G2 lens 14 and the G3 lens 15 .

The eyepiece window 17 is a transparent member that transmits visible light. An image displayed on the display panel unit 8 is observed through the optical path splitting prism unit 11 , eyepiece optical system 16 and eyepiece window 17 .

The illumination windows 20, 21 are windows for hiding the infrared LEDs 18, 19, 22 to 27 from being visible from the outside, and are made of a resin that absorbs visible light and transmits infrared light.

4(a) is a perspective view showing the configuration of the EVF portion of the camera 1, FIG. 4(b) is a side view showing the configuration of the EVF portion near infrared LEDs, and FIG. 4(b) is the optical axis of the EVF portion. is a cross-sectional view of the.

The infrared LEDs 18 , 19 , 22 , 23 , 24 , 25 , 26 , 27 are arranged at different positions and orientations so that each of them emits light (infrared light) toward the peephole 12 . The infrared LEDs 18, 19, 23, and 25 are infrared LEDs (light sources) for short-distance illumination, and emit infrared light with a first wavelength λ1. The infrared LEDs 22, 24, 26, and 27 are infrared LEDs (light sources) for long-distance illumination, and emit infrared light with a second wavelength λ2 longer than the first wavelength λ1. Light sources other than infrared LEDs may be used.

A line-of-sight detection optical system including an aperture 28 and a line-of-sight imaging lens 29 guides reflected infrared light guided from the eyepiece window 17 by the optical path splitting prism unit 11 to a line-of-sight detection sensor 30 .

The line-of-sight detection sensor 30 is composed of a solid-state imaging device (imaging sensor) such as a CCD or CMOS. The infrared LEDs 18, 19, 22 to 27 irradiate the user's eyeballs with infrared light. Receives reflected light emitted from and reflected by the eyeball).

In FIG. 4B, the angle θ1 is the infrared LEDs 18, 19, 23, 25 for short-distance illumination.
, and the optical axis LF of the eyepiece optical system 16. The angle θ2 is the angle between the irradiation direction L2 of the infrared LEDs 22, 24, 26, and 27 for long-distance illumination and the optical axis LF of the eyepiece optical system 16. FIG. And the angle θ1 is larger than the angle θ2. By doing so, each of the infrared LEDs 18, 19, 22 to 27 can irradiate the user's eyeball with infrared light at an angle with a high irradiation intensity (radiation intensity). And the efficiency of the irradiation intensity to the eyeball can be improved with respect to the power consumption of the infrared LED. For example, the angle θ1=30° and the angle θ2=20°.

Here, consider the case where the eyeball of the user looking into the finder is irradiated with infrared light from at least one of the infrared LEDs 18, 19, 22-27. As shown by the optical path 31a in FIG. 4(c), an optical image (eyeball image) of an eyeball irradiated with infrared light passes through the eyepiece window 17, the G3 lens 15, the G2 lens 14, and the G1 lens 13, and passes through the second lens. It enters the second optical path splitting prism 10 from the second surface 10 a of the optical path splitting prism 10 . A dichroic film that reflects infrared light is formed on the first surface 10b of the second optical path splitting prism 10. As shown by the reflected light path 31b, the eyeball image entering the second optical path splitting prism 10 is The first surface 10b reflects toward the second surface 10a. Then, as indicated by the imaging optical path 31c, the reflected eyeball image is totally reflected by the second surface 10a, and exits the second optical path splitting prism 10 from the third surface 10c of the second optical path splitting prism 10, The light passes through the diaphragm 28 and is imaged in the vicinity of the line-of-sight detection sensor 30 by the line-of-sight imaging lens 29 . For line-of-sight detection, a corneal reflection image formed by regular reflection of infrared light emitted from an infrared LED on the cornea is used together with such an eyeball image. The vicinity of the line-of-sight detection sensor 30 includes above the line-of-sight detection sensor 30 .

FIG. 5 shows an example of an optical path from the infrared light emitted from the infrared LEDs 18, 19, 23, 25 for short-distance illumination to specular reflection by the cornea 37 of the eyeball and reception by the line-of-sight detection sensor 30. .

<Description of gaze detection operation>
A line-of-sight detection method will be described with reference to FIGS. FIG. 6 is a diagram for explaining the principle of the line-of-sight detection method, and is a schematic diagram of an optical system for performing line-of-sight detection. As shown in FIG. 6, the infrared LEDs 18 and 25 illuminate the user's eyeball 140 with infrared light. Part of the infrared light emitted from the infrared LEDs 18 and 25 and reflected by the eyeball 140 is imaged near the line-of-sight detection sensor 30 by the line-of-sight imaging lens 29 . In FIG. 6, the positions of the infrared LEDs 18 and 25, the line-of-sight imaging lens 29, and the line-of-sight detection sensor 30 are adjusted so that the principle of the line-of-sight detection method can be easily understood. Although the case where the infrared LEDs 18 and 25 emit infrared light has been described as an example, the same applies to cases where the other infrared LEDs 19, 22, 23, 24, 26, and 27 emit infrared light. .

FIG. 7A is a schematic diagram of an eye image captured by the line-of-sight detection sensor 30 (an eyeball image projected on the line-of-sight detection sensor 30), and FIG. FIG. 4 is a diagram showing output intensity; FIG. 8 shows a schematic flow chart of the line-of-sight detection operation. FIGS. 9(a) to 9(c) are diagrams for explaining the relationship between the wavelength of the infrared LED and the imaging state of the eye image according to the first embodiment.

When the line-of-sight detection operation starts, the CPU 3 detects the eyeball distance s in step S801 of FIG. In the first embodiment, the CPU 3 detects the eyeball distance s based on an input from the user. For example, the CPU 3 displays on the display device 209 that the user is asked whether or not he/she wears glasses. The user uses the operation members 42 and 43 to input to the camera 1 whether or not to wear eyeglasses. When the user inputs that the eyeglasses are not worn, the CPU 3 determines that the eyeball distance s is 30 mm (detects the eyeball distance s=30 mm). On the other hand, when the user inputs that the eyeglasses are worn, the CPU 3 determines that the eyeball distance s is 50 mm (detects the eyeball distance s=50 mm). The eyeball distance s is the distance from the camera 1 to the eyes of the user, for example, the distance from the eyepiece to the eyes. In the first embodiment, the distance from the line-of-sight imaging lens 29 to the eye (the distance along the optical axis of the line-of-sight imaging lens 29; the shortest optical path length) is defined as the eyeball distance s.

In step S802, CPU 3 determines an infrared LED to be used for line-of-sight detection. In the first embodiment, the CPU 3 selects an infrared LED for short-distance lighting and an infrared LED for long-distance lighting as the infrared LEDs used for visual line detection according to the line-of-sight distance s detected in step S801. switch between Specifically, in step S802, the CPU 3 determines whether or not the eyeball distance s is equal to or less than a predetermined distance (threshold value) of 40 mm. When the CPU 3 determines that the eyeball distance s is equal to or less than the predetermined distance (threshold value) 40 mm, the CPU 3 determines the infrared LED for short-distance lighting as the infrared LED to be used for line-of-sight detection, and advances the process to step S803. . If the CPU 3 determines that the eyeball distance s is longer than the predetermined distance 40 mm, the CPU 3 determines the infrared LED for far and near distance illumination as the infrared LED to be used for line-of-sight detection, and advances the process to step S804.

In step S<b>803 , CPU 3 drives two of infrared LEDs 18 , 19 , 23 , and 25 for short-distance lighting via light source driving section 205 . In this case, two of the infrared LEDs 18 , 19 , 23 , 25 for short-distance illumination emit infrared light toward the user's eyeball 140 . The remaining infrared LEDs (infrared LEDs 22, 24, 26, 27 for long-distance illumination) are extinguished. The user's eyeball image illuminated by the infrared light is formed in the vicinity of the line-of-sight detection sensor 30 through the line-of-sight imaging lens 29 (light receiving lens), and photoelectrically converted by the line-of-sight detection sensor 30 . This provides an electrical signal of the eye image that can be processed.

In step S<b>804 , CPU 3 drives two of the infrared LEDs 22 , 24 , 26 , 27 for long-distance lighting via light source driving section 205 . In this case, two of the infrared LEDs 22 , 24 , 26 , 27 for long-distance illumination emit infrared light toward the user's eyeball 140 . The rest of the infrared LEDs (infrared LEDs 18, 19, 23, 25 for near-field illumination) are extinguished. As described above, the user's eyeball image illuminated by the infrared light is formed in the vicinity of the line-of-sight detection sensor 30 through the line-of-sight imaging lens 29 and photoelectrically converted by the line-of-sight detection sensor 30 . This provides an electrical signal of the eye image that can be processed.

Here, the relationship between the wavelength of the infrared LED and the imaging state of the eye image will be described with reference to FIGS. 9(a) to 9(c). 9(a) to 9(c) are conceptual diagrams of the line-of-sight detection mechanism, which is a straight system in which the line-of-sight imaging lens 29 and the line-of-sight detection sensor 30 are arranged at the position where the reflection by the second optical path splitting prism 10 is developed. It is a sectional view. The illustration of members such as the eyepiece optical system 16 is omitted.

FIG. 9A shows an eyeball image formation state when the eyeball distance s is short and the eye is irradiated with infrared light of the first wavelength λ1 by the infrared LED for short-distance illumination. The case where the eyeball distance s is short is, for example, the case where the eyeglasses are not worn. In this case, the line-of-sight imaging lens 29 forms a pupil image on the line-of-sight detection sensor 30 . Therefore, an in-focus eye image can be obtained, and highly accurate line-of-sight detection is possible.

FIG. 9(b) shows the formed state of the eyeball image when the eyeball distance s is long and the eye is irradiated with the infrared light of the first wavelength λ1 by the infrared LED for short-distance illumination. A case where the eyeball distance s is long is, for example, a case where spectacles are worn. In the case of FIG. 9B, the line-of-sight imaging lens 29 forms the pupil image in front of the line-of-sight detection sensor 30 according to Newton's imaging formula. . As a result, an out-of-focus eye image is obtained, and the line-of-sight detection accuracy is lowered.

Therefore, in the first embodiment, as shown in FIG. 9C, when the eyeball distance s is long, the eyeball is irradiated with the second wavelength λ2 longer than the first wavelength λ1. FIG. 9(c) shows the formed state of the eyeball image when the eyeball distance s is long and the eye is irradiated with the light of the second wavelength λ2 by the infrared LED for long-distance illumination. Glass and resin, which are generally used as materials for the line-of-sight imaging lens 29, have optical characteristics such that the longer the wavelength, the smaller the refractive index. Therefore, by lengthening the wavelength of the light incident on the line-of-sight imaging lens 29 and decreasing the corresponding refractive index, the imaging position of the pupil image moves to the back. Thus, even when the eyeball distance s is long, the line-of-sight imaging lens 29 can form the pupil image on the line-of-sight detection sensor 30 . As a result, an in-focus eye image is obtained, and highly accurate line-of-sight detection becomes possible.

Here, it is assumed that the near eyeball distance s1 is 30 mm and the far eyeball distance s2 is 50 mm. Also, the line-of-sight imaging lens 29 is assumed to be a thin spherical lens. Assume that the radius r2 is -1 mm. It is assumed that the distance s' between the line-of-sight imaging lens 29 and the line-of-sight detection sensor 30 is 0.8 mm. It is assumed that the line-of-sight imaging lens 29 has optical characteristics of refractive index N1=1.570 corresponding to wavelength λ=850 nm and refractive index N2=1.561 corresponding to wavelength λ=1150 nm. In this case, by setting the first wavelength λ1 to 850 nm and the second wavelength λ2 to 1150 nm, an in-focus eyeball image can be obtained regardless of whether the eyeball distance s is short or long, and a highly accurate line of sight can be obtained. Detection becomes possible.

Note that specific values such as the first wavelength λ1 and the second wavelength λ2 are not limited to the values described above, and when a single wavelength is used by changing the wavelength when the eyeball distance s changes Any value may be used as long as the imaging state is improved compared to .

However, considering the general optical characteristics and shape of the line-of-sight imaging lens 29 and the general spectral sensitivity characteristics of the line-of-sight detection sensor 30, the first wavelength λ1 is preferably 700 nm or more and 1000 nm or less. The second wavelength λ2 is preferably 900 nm or more and 1200 nm or less. In addition, the first wavelength λ1 is preferably determined in consideration of the eyeball distance s, etc., which is statistically likely to occur in users wearing eyeglasses, and the second wavelength λ2 is not wearing eyeglasses. It is preferable to determine the eyeball distance s, which is statistically likely to occur in the user, and the like. Furthermore, when the first wavelength λ1 and the second wavelength λ2 are wavelengths in the visible light region, the light with the first wavelength λ1 and the light with the second wavelength λ2 are displayed on the second display panel 6. This may affect the visibility when the user sees the image. Therefore, the first wavelength λ1 and the second wavelength λ2 are preferably wavelengths in the invisible light region (800 nm or longer).

In step S<b>805 , the line-of-sight detection unit 201 (line-of-sight detection circuit) sends the eye image (eye image signal; electric signal of the eye image) obtained from the line-of-sight detection sensor 30 to the CPU 3 .

In step S806, CPU 3 obtains the coordinates of points corresponding to the corneal reflection images Pd and Pe of the two infrared LEDs driven in step S803 or step S804 and the pupil center c from the eye image obtained in step S805. .

Infrared light emitted by the two infrared LEDs illuminates the cornea 142 of the user's eyeball 140 . At this time, the corneal reflection images Pd and Pe formed by part of the infrared light reflected on the surface of the cornea 142 are collected by the line-of-sight imaging lens 29 and formed in the vicinity of the line-of-sight detection sensor 30 to form an image of the eye. These are corneal reflection images Pd' and Pe' in the image. Similarly, light from the ends a and b of the pupil 141 is also imaged in the vicinity of the line-of-sight detection sensor 30 to form pupil end images a' and b' in the eye image.

FIG. 7(b) shows luminance information (luminance distribution) of the region α' in the eye image of FIG. 7(a).
FIG. 7B shows the luminance distribution in the X-axis direction, with the horizontal direction of the eye image being the X-axis direction and the vertical direction being the Y-axis direction. In the first embodiment, the X-axis (horizontal) coordinates of the corneal reflection images Pd' and Pe' are Xd and Xe, and the X-axis coordinates of the pupil edge images a' and b' are Xa and Xb. do. As shown in FIG. 7B, extremely high levels of luminance are obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from the coordinate Xa to the coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by forming an image of the light from the pupil 141 near the line-of-sight detection sensor 30), except for the coordinates Xd and Xe, Extremely low levels of brightness are obtained. In the area of the iris 143 outside the pupil 141 (the area of the iris image outside the pupil image obtained by forming an image of the light from the iris 143), a brightness intermediate between the above two types of brightness is obtained. Specifically, a luminance intermediate between the above two types of luminance is obtained in an area where the X coordinate (coordinate in the X-axis direction) is smaller than the coordinate Xa and in an area where the X coordinate is larger than the coordinate Xb.

From the luminance distribution as shown in FIG. 7B, the X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil edge images a' and b' can be obtained. Specifically, the coordinates with extremely high brightness can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and the coordinates with extremely low brightness can be obtained as the coordinates of the pupil edge images a' and b'. . Further, when the rotation angle θx of the optical axis of the eyeball 140 with respect to the optical axis of the line-of-sight imaging lens 29 is small, the light from the pupil center c forms an image in the vicinity of the line-of-sight detection sensor 30 to form a pupil center image c′. The coordinate Xc of (the center of the pupil image) can be expressed as Xc≈(Xa+Xb)/2. That is, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

In step S807, the CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is a magnification determined by the position of the eyeball 140 with respect to the line-of-sight imaging lens 29, and can be obtained using the function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

The longer the eyeball distance s or the shorter the interval between the infrared LEDs, the shorter the interval between the corneal reflection images Pd′ and Pe′. There is a possibility that the resolution will be lowered. In the first embodiment, on a plane perpendicular to the optical axis of the eyepiece optical system 16, the distance from the optical axis of the eyepiece optical system 16 to the infrared LEDs 22, 24, 26, and 27 for long-distance illumination is 16 from the optical axis to the infrared LEDs 18, 19, 23, 25 for short-distance illumination. In other words, the infrared LEDs 22, 24, 26, 27 for long-distance illumination are arranged outside the infrared LEDs 18, 19, 23, 25 for short-distance illumination with respect to the optical axis of the eyepiece optical system 16. ing. As a result, even when the eyeball distance s is long, it is possible to secure a certain distance between the corneal reflection images Pd' and Pe'. As a result, the reduction in the detection resolution of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe' by the infrared LED for long-distance illumination is suppressed, and the image formation of the eyeball image calculated from the interval (Xd-Xe) By suppressing the deterioration of the accuracy of the magnification β, it is possible to suppress the deterioration of the sight line detection accuracy.

In step S<b>808 , CPU 3 calculates the rotation angle of the optical axis of eyeball 140 with respect to the optical axis of line-of-sight imaging lens 29 . The X coordinate of the midpoint between the corneal reflection image Pd and the corneal reflection image Pe and the X coordinate of the center of curvature O of the cornea 142 substantially match. Therefore, if the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Oc, then the rotation angle θx of the eyeball 140 in the ZX plane (the plane perpendicular to the Y-axis) is given below. It can be calculated by the formula 1 below. The rotation angle θy of the eyeball 140 within the ZY plane (the plane perpendicular to the X axis) can also be calculated by a method similar to the method for calculating the rotation angle θx.
β×Oc×SINθx≈{(Xd+Xe)/2}−Xc (Formula 1)

In step S809, the CPU 3 uses the rotation angles θx and θy calculated in step S808 to determine the user's line-of-sight position (the position where the user's line of sight is directed) in the image for visual recognition displayed on the second display panel 6. position) is obtained (estimated). Coordinates of line-of-sight position (Hx
, Hy) are the coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the line-of-sight position can be calculated by the following equations 2 and 3.
Hx=m×(Ax×θx+Bx) (Formula 2)
Hy=m×(Ay×θy+By) (Formula 3)

The parameter m in Equations 2 and 3 is a constant determined by the configuration of the viewfinder optical system (line-of-sight imaging lens 29, etc.) of the camera 1, and the rotation angles θx and θy are converted to coordinates corresponding to the pupil center c in the image for visual recognition. , which is determined in advance and stored in the memory unit 4 . The parameters Ax, Bx, Ay, and By are line-of-sight correction parameters for correcting individual differences in line-of-sight, are obtained by performing a known calibration work, and are stored in the memory unit 4 before the line-of-sight detection operation is started. and

In step S810, the CPU 3 stores the line-of-sight position coordinates (Hx, Hy) in the memory unit 4, and ends the line-of-sight detection operation.

Note that the line-of-sight detection method is not limited to the above method. More or less than two infrared LEDs may be used for line-of-sight detection.

As described above, according to the first embodiment, the light sources (infrared LEDs) used for line-of-sight detection are a first light source that emits light of a first wavelength and a second light source that emits light of a second wavelength. Switchable between 2 light sources. Specifically, the light source used for line-of-sight detection is switched according to the detected eyeball distance. As a result, the imaging state of the pupil image in the eye image is improved as compared with the case of using light of a single wavelength. Therefore, it is possible to provide an electronic device capable of highly accurate line-of-sight detection regardless of eyeball distance.

Note that the eyeball distance detection method is not limited to a method based on user input. For example, the CPU 3 detects the eyeball distance based on the user's input in step S801 in the first sight line detection operation, stores the detected eyeball distance in the memory unit 4, and in step S801 after the second time, the memory unit 4, the eyeball distance may be read. Further, in step S801, the CPU 3 receives the reflected light when the user's eyes are irradiated with the infrared light from the infrared LED for short-distance lighting and/or the infrared LED for long-distance lighting. Based on the output of 30, eye distance may be detected. For example, the CPU 3 detects an eye image obtained by the line-of-sight detection sensor 30 while the user's eyes are irradiated with infrared light from at least one of the infrared LED for short-distance lighting and the infrared LED for long-distance lighting. You may detect eyeball distance based on. Since the average brightness of the entire eye image decreases as the eyeball distance increases, it is possible to detect the eyeball distance based on the average brightness. For example, the CPU 3 may acquire an eye image by performing processing similar to steps S803 and S805, and detect the eyeball distance based on the average brightness of the eye image. The eyeball distance may be detected based on an eye image acquired using one infrared LED, or multiple infrared LEDs, each of which is an infrared LED for short-distance illumination or an infrared LED for long-distance illumination, may be used. The eyeball distance may be detected based on the eye image acquired using the LED. Since the longer the eyeball distance, the shorter the interval between the plurality of corneal reflection images corresponding to the plurality of infrared LEDs, the eyeball distance can be detected based on the interval. For example, the CPU 3 performs the same processing as in steps S803 and S805 to acquire an eye image, performs the same processing as in S806 to detect a plurality of corneal reflection images from the eye image, and extracts the detected corneal reflection images. Eyeball distance may be detected based on the distance.

[Second embodiment]
A second embodiment of the present invention will be described below. FIG. 10 represents a schematic flow chart of the line-of-sight detection operation according to the second embodiment. In the second embodiment, matters other than the line-of-sight detection operation are the same as those in the first embodiment. In the first embodiment, the infrared LED used for line-of-sight detection is switched according to the detected eyeball distance. In the second embodiment, when line-of-sight detection using an infrared LED for short-distance lighting or an infrared LED for long-distance lighting fails, the infrared LED used for line-of-sight detection is switched.

10, step S1001 is the same as step S803 in FIG. 8, steps S1002 and S1006 are the same as step S805, and steps S1003 and S1007 are the same as step S806. Step S1005 is the same as step S804, and steps S1008 to S1011 are the same as steps S807 to S810.

In step S1004, the CPU 3 determines whether or not the pupils (the coordinates of the pupil edge images a' and b' and the pupil center image c') could be detected in step S1003. If the pupil cannot be detected, the line-of-sight detection using the infrared LED for short-distance illumination fails. Here, "failure" includes not only the inability to detect the line of sight, but also the inability to detect the line of sight with high accuracy. When the CPU 3 determines that the pupil has been detected, the process proceeds to step S1008. If the CPU 3 determines that the pupil could not be detected, the process advances to step S1005 (the infrared LED used for line-of-sight detection changes from the infrared LED for short-distance illumination to the infrared LED for long-distance illumination). ).

In the second embodiment, the processing in step S801 (detection of eyeball distance) and the processing in step S802 (determination of infrared LEDs for line-of-sight detection) performed in the first embodiment are not performed. In addition, an infrared LED for short-distance illumination is used as an infrared LED for line-of-sight detection. Therefore, it is suitable for users whose eyeball distance is short. In the case of a user whose eyeball distance is short, the processing of steps S1005 to S1007 is not performed, and the time for the line-of-sight detection operation can be shortened by the processing time of steps S801 and S802 compared to the first embodiment. . In addition, it is possible to save the trouble of inputting by the user in step S801.

On the other hand, for a user whose eyeball distance is long (when it is determined in step S1004 that the pupil could not be detected), the first embodiment is more suitable. In the second embodiment, in the case of a user with a long eyeball distance, the operations from driving the infrared LEDs to acquiring the coordinates of the corneal reflection images Pd′ and Pe′ and the pupil center image c′ are performed in steps S1001 to S1003. , steps S1005 to S1007 must be performed twice. Therefore, the time required for the line-of-sight detection operation is longer than in the first embodiment.

Although an example in which the infrared LED for short-distance illumination is first used has been described, the infrared LED for long-distance illumination may be used first. When line-of-sight detection using an infrared LED for long-distance illumination fails, the infrared LED used for line-of-sight detection is switched from an infrared LED for long-distance illumination to an infrared LED for short-distance illumination. may

The above-described embodiment (including modifications) is merely an example, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the present invention are also included in the present invention. . A configuration obtained by appropriately combining the configurations described above is also included in the present invention.

<Other embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

1: Camera 3: CPU 30: Line-of-sight detection sensor 18, 19, 22-27: Infrared LED

Claims (16)

An electronic device capable of executing line-of-sight detection for detecting a user's line of sight,
a first light source that irradiates the eye of the user with light of a first wavelength;
a second light source that irradiates the eye of the user with light of a second wavelength different from the first wavelength;
a sensor that receives light for detecting the line of sight;
An electronic device, comprising control means for switching a light source used for detecting the line of sight between the first light source and the second light source. The distance from the electronic device to the user's eye is detected based on the output of the sensor that receives the reflected light when the user's eye is irradiated with the light from the first light source and/or the second light source. further comprising a detection means for
2. The electronic device according to claim 1, wherein said control means switches between said first light source and said second light source to use for said line of sight detection according to the distance detected by said detection means. machine. 3. The electronic device according to claim 2, wherein said detection means detects a distance from an eyepiece of said electronic device to a user's eye. the second wavelength is longer than the first wavelength;
The control means is
determining the first light source as a light source to be used for the line-of-sight detection when the detected distance is shorter than a threshold;
4. The electronic device according to claim 2, wherein when the detected distance is longer than the threshold, the second light source is determined as the light source used for the line-of-sight detection. When the distance from the electronic device to the eyes of the user is input by the user, the control means selects one of the first light source and the second light source according to the distance to be used for the line-of-sight detection. 2. The electronic device according to claim 1, wherein the switching is performed between . the sensor is an imaging sensor;
The detection means detects the distance based on an average luminance of an image obtained by the sensor while the eyes of the user are irradiated with light from at least one of the first light source and the second light source. The electronic device according to any one of claims 2 to 4, characterized by: having a plurality of light sources, each being said first light source or said second light source;
the sensor is an imaging sensor;
The detection means, based on intervals between a plurality of corneal reflection images respectively corresponding to the plurality of light sources in an image obtained by the sensor while the eyes of the user are irradiated with light from the plurality of light sources, 5. The electronic device according to claim 2, wherein said distance is detected. 2. The electronic device according to claim 1, wherein the control means switches the light source used for the line-of-sight detection when the line-of-sight detection using the first light source or the second light source fails. the sensor is an imaging sensor;
The control means detects the line of sight when the eye of the user is irradiated with light from at least one of the first light source and the second light source and the pupil is not detected from the image obtained by the sensor. 9. The electronic device according to claim 1, wherein the light source used for the electronic device is switched. 10. The electronic device according to claim 1, wherein the first wavelength is 700 nm or more and 1000 nm or less. The electronic device according to any one of claims 1 to 10, wherein the second wavelength is 900 nm or more and 1200 nm or less. The angle formed by the irradiation direction of the first light source and the optical axis of the optical system provided in the portion where the user eyepieces is larger than the angle formed by the irradiation direction of the second light source and the optical axis of the optical system. The electronic device according to any one of claims 1 to 11, characterized in that it is large. The distance from the optical axis of the optical system to the second light source is the distance from the optical axis of the optical system to the first light source in a plane perpendicular to the optical axis of the optical system provided in the portion where the user eyepieces. The electronic device according to any one of claims 1 to 12, wherein the distance is longer than the distance. imaging means;
The electronic device according to any one of claims 1 to 13,
An imaging device characterized by comprising: a first light source that irradiates a user's eye with light of a first wavelength;
a second light source that irradiates the eye of the user with light of a second wavelength different from the first wavelength;
A control method for an imaging device having a sensor that receives light for detecting a line of sight of the user,
switching the light source used for the line-of-sight detection between the first light source and the second light source;
and a step of performing the line-of-sight detection. A program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 13.

Images