KR20260014646A - Eye tracking system and method - Google Patents

Abstract

The present invention relates to an eye tracking system (100). The system comprises at least one projection means (3) configured to project a light beam (4) onto a predetermined area of the cornea (2) of an eye (1). The system further comprises at least one optical sensor (6) capable of detecting light (5) reflected from the eye (1) and determining a detection position thereon. The light (5) reflected from the eye (1) comprises at least one, preferably two, specular reflections, preferably at least a first Purkinje reflection (P1), more preferably at least a first Purkinje reflection (P1) and a fourth Purkinje reflection (P4). The system (100) is configured to calculate a lateral movement and/or an angular movement or a change thereof of the eye (1) based on the detection position and/or the displacement thereof.

Description

Eye tracking system and method

The present invention relates to the field of optical sensing systems, and more particularly to systems and methods for eye tracking.

Optical sensing systems are used in a variety of applications, one of which is eye tracking, which is particularly important in augmented reality (AR), virtual reality (VR), and mixed reality (XR). In many cases, these systems are positioned very close to the user's eyes, for example, on a pair of eyeglasses or a head-mounted system. These systems typically operate by emitting light of a specific wavelength over the entire area of the eye, observing and analyzing the reflected light. This is challenging because the optical power of the emitted light must be very low to ensure eye safety. However, developing such systems is challenging, especially since the area under the eyeglasses (e.g., VR glasses) is typically dark. This means the system must operate in a dark environment, meaning it must identify its own light sources without relying on other light sources in the environment, distinguishing them from other ambient reflections. Additionally, the signal-to-noise ratio must be as high as possible. Another area for improvement is the accuracy of the system. On the other hand, making the system smaller also reduces accuracy, as small changes in eye orientation become harder to detect. Furthermore, such a system must be power-efficient and capable of detecting eye orientation with very low latency.

Therefore, there is a need for optical sensing systems that enable accurate, low-power, low-latency, and safe eye tracking. The present invention relates to the eye and aims to address or at least improve upon some of the problems mentioned above.

In a first aspect, the present invention relates to an optical sensing system for 3D imaging. The system comprises at least one projection means configured to project a projection output onto a predetermined area of the cornea of an eye. The system further comprises at least one optical sensor capable of detecting a reflected light from the eye and determining a detection position thereon. The system is configured to calculate a lateral translation and/or an angular translation of the eye and/or a change thereof based on the detection position and/or the displacement thereof. The reflected light from the eye comprises at least one reflection, preferably two reflections, preferably at least a first Purkinje reflection, more preferably at least a first Purkinje reflection and a fourth Purkinje reflection.

Preferred embodiments of the first aspect of the present invention comprise one or more of the following features or a suitable combination thereof:

- wherein said at least one projection means is a single movable projection means positioned such that said at least one reflection is received by said optical sensor, or wherein said at least one projection means is a plurality of projection means, preferably arranged in a one-dimensional or two-dimensional array, and wherein said system is configured to find at least one projection means that causes said at least one reflection to be received by said optical sensor, preferably at least a first Purkinje reflection and a fourth Purkinje reflection.

- Among the plurality of projection means, at least one projection means is switched to another projection means based on the detection position and/or its displacement.

- Each photodetector is a single photon avalanche detector (SPAD) that can detect single photons.

- The optical sensor comprises a plurality of sensing units, preferably forming a matrix of pixels, each sensing unit comprising a photodetector.

- The system is configured to filter a detection location determined by one of the plurality of sensing units based on the presence of the detection on at least one neighboring sensing unit adjacent to the one sensing unit, or based on the presence of the detection over at least two consecutive time steps.

- Said at least one optical sensor has a minimum field of view of at least 5 millimeters on the eye image plane along the horizontal axis of the eye, preferably at least 10 millimeters, more preferably at least 15 millimeters, said field of view being covered by one optical sensor having a full field of view at least corresponding to said minimum field of view, or by one movable optical sensor whose position is movable to have a full field of view at least corresponding to said minimum field of view, or by a plurality of optical sensors having a full field of view at least corresponding to said minimum field of view.

- The projection means is configured to be reoriented so that the projection output is maintained in the predetermined area of the cornea when the eye rotates, the reorientation being based on the detection position Δ and/or its displacement.

- The system further comprises reflecting means, preferably dynamic deflection mirrors, more preferably MEMS, wherein the projection output is projected onto the reflecting means such that the reflecting means reflects the projection output onto the predetermined area of the cornea, the reflecting means being configured to be reoriented so that when the eye rotates, the projection output remains in the predetermined area of the cornea, the reorientation being based on the detection position and/or displacement thereof.

- The system is configured to find at least one different projection means among the plurality of projection means, wherein the different projection means causes a fourth Purkinje reflection to be received by the optical sensor.

- The above system is calibrated by scanning the projection output of the projection means along a predetermined scanning trajectory on the cornea.

- The at least one optical sensor is configured to measure the position or positions of the at least one reflection and provides the position or positions in an output of the sensor.

- The at least one optical sensor comprises a first optical sensor and a second optical sensor, each optical sensor being configured to receive different reflections.

- The above optical sensor is positioned on the conjugate plane of the pupil.

- The projection means is operated in a pulsed mode, each pulse having a first predetermined intensity or a second predetermined intensity, or each pulse having a first predetermined pulse width or a second predetermined pulse width, wherein the first predetermined intensity or the first predetermined pulse width is configured to cause one reflection to be received by the optical sensor, and the second predetermined intensity or the second predetermined pulse width is configured to cause one different reflection to be received by the optical sensor.

- The at least one optical sensor is configured to simultaneously measure at least two, preferably two, detection positions on the optical sensor, providing the positions at an output of the sensor, each of the positions corresponding to a different specular reflection.

In a second aspect, the present invention relates to a display system using an eye tracking system according to the first aspect.

In a third aspect, the present invention relates to a method for tracking eye orientation. The method comprises the following steps:

- illuminating the cornea of the eye to be traced with a light beam focused on a predetermined area of the cornea, preferably in a pulsed mode, by means of a projection means;

- A step of detecting a reflection from an eye by at least one optical sensor, wherein the reflection comprises at least one, preferably two, specular reflections, the reflections being preferably at least a first Purkinje reflection, more preferably at least a first Purkinje reflection and a fourth Purkinje reflection;

- a step of determining a detection position on at least one optical sensor;

- A step of calculating the orientation and/or position of the eye (e.g., the transverse position and/or angular position of the eye) and/or the change thereof from the determined detection position and/or the displacement thereof.

Preferred embodiments of the third aspect of the present invention comprise one or more of the following features or a suitable combination thereof:

o the projection means is a single movable projection means, and the method comprises the step of positioning the projection means so that the reflected light is received by the optical sensor; or

o The projection means are a plurality of projection means arranged in a one-dimensional or two-dimensional array, and the method includes a step of finding at least one projection means that causes the reflected light to be received by the optical sensor,

The method comprises a step of switching operation from one projection means to another projection means based on the detection position and/or displacement thereof in the plurality of projection means so that the reflected light is received by the optical sensor,

- The optical sensor comprises a plurality of sensing units forming a matrix of pixels, each sensing unit comprising a photodetector, preferably a single-photon avalanche detector,

- The method comprises a step of filtering a detection location determined by one of the plurality of sensing units based on the presence of the detection on at least one neighboring sensing unit adjacent to the one sensing unit, or based on the presence of the detection over at least two consecutive time steps,

- The method further comprises the step of configuring the at least one optical sensor to have a minimum field of view of at least 5 millimeters on the eye image plane along the horizontal axis of the eye, preferably at least 10 millimeters, more preferably at least 15 millimeters, wherein the field of view is:

o one optical sensor having a full field of view corresponding to at least the minimum field of view; or

o one movable optical sensor capable of being moved to have a full field of view corresponding to at least the minimum field of view; or

o A plurality of optical sensors having a full field of view corresponding to at least the minimum field of view mentioned above.

Covered by,

- The method further comprises a step of reorienting the projection means based on the detection position and/or its displacement so that the light beam remains in the predetermined area of the cornea when the eye rotates.

- The above method:

o A step of projecting the above light beam onto a reflecting means;

o A step of reflecting the light beam onto the predetermined area of the cornea;

o a step of reorienting the reflecting means so that the light beam remains in the predetermined area of the cornea when the eye rotates based on the detection position and/or its displacement;

Including more,

- The method further comprises the step of finding at least one different projection means among the plurality of projection means that causes the fourth Purkinje reflection to be received by the optical sensor,

- The method comprises a step of scanning a light beam along a predetermined scanning trajectory on the cornea,

- wherein said at least one optical sensor comprises a first optical sensor and a second optical sensor, and said method comprises:

o Configuring a first optical sensor to receive one reflection; and

o A step of configuring a second optical sensor to receive one different reflection.

Including more,

- The method includes a step of positioning the optical sensor on the standard plane of the pupil,

- The above method:

o A step of operating the projection means in a pulsed mode;

o Steps to configure each pulse to have:

■ First predetermined intensity or second predetermined intensity, or

■ First predetermined pulse width or second predetermined pulse width;

o a step of adapting a first predetermined intensity or a first predetermined pulse width such that one reflection is received by the optical sensor; and

o a step of adapting a second predetermined intensity or a second predetermined pulse width such that one different reflection is received by the optical sensor;

Includes.

The above and other features, characteristics, and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate the principles of the present invention by way of example. This description is provided solely for illustrative purposes and is not intended to limit the scope of the invention.

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate several aspects of the invention and, together with the description, serve to explain the principles of the invention. In the drawings:
Figure 1 schematically illustrates an eye tracking system (100) according to the present invention.
Figure 2 illustrates different layers (41-44) of the eye (1) and different Purkinje reflexes (P1-P4) according to the present invention.
Figures 3 and 4 schematically illustrate alternative configurations for the eye tracking system (100) according to the present invention.
FIG. 5 illustrates, according to the present invention, reflective means (10) for focusing light on different parts of the eye (1) in (a, b) and an extended depth of focus obtained by having a plurality of projection means (3) in (c, d).
Figure 6 illustrates an embodiment of an optimized eye tracking system according to the present invention.
Figure 7 illustrates another embodiment of a system according to the present invention.
FIG. 8 illustrates an embodiment of a system similar to FIG. 7, which considers only one transverse position of the eye according to the present invention.
FIG. 9 illustrates an embodiment of a system in which different z-positions of the eye are taken into account according to the present invention.
While the present invention will be described with reference to specific preferred embodiments, it is not intended to be limited thereto. Rather, the intention is to cover all alternatives, modifications, and equivalents falling within the spirit and scope of the invention as defined by the appended claims.

In a first aspect, the present invention relates to an eye tracking system for eye tracking. The system comprises at least one projection means configured to project a projection output (e.g., a light beam) onto a predetermined portion (e.g., an area) of the cornea of the eye. For example, the projection means is a light source, preferably a laser light source capable of generating laser output. For example, the projection means is a light source having a substantially small numerical aperture, such that the image of the light source on the eye is substantially small. The use of a laser light source is advantageous due to the small numerical aperture, as it allows for accommodating different users with different eye position distances from the face. For example, people may have different nose sizes and different eye depths, which may cause the system to be moved further or closer to the eyes, but not closer than a minimum distance determined by the design for eye relief, such as 10 mm. In other words, the projector should ensure that the Rayleigh length of the projected beam remains within the depth of focus of the imaging system, regardless of the distance at which the eyes are positioned along the z-axis. For example, the total angular spread of the beam, which is inversely proportional to the Rayleigh length, should be as small as possible. For example, the Rayleigh length can be greater than 5 mm, and preferably greater than 10 mm.

Prior art systems have a wide beam that covers the entire eye. While this wide beam ensures that reflections from the eye are captured, it is power-inefficient. In the present invention, the projection means generates a narrow beam that covers a small portion of the eye, while still generating the necessary reflections, thereby making the system power-efficient. For example, the projection output on the eye is a point-shaped output, such as one or more points on the eye. For example, the beam has a minimum spot size necessary to generate the necessary reflections, and for example, the reflections received on the optical sensor are identifiable and do not overlap with each other. For example, the numerical aperture must be sufficiently large to prevent reflections from overlapping over the tracking range.

The system further comprises at least one optical sensor capable of detecting light reflected from the eye and determining a detection location thereon. For example, the projection means and the optical sensor are positioned such that the projection output is incident on the eye, reflected, and received by the optical sensor. The optical sensor can then determine the detection location, which is associated with the position of the projection means and the angle at which it is positioned.

The output of the above projection means is preferably pulsed, for example having a pulse width of at least 1 nanosecond, and preferably at least 10 nanoseconds. Preferably, the pulse width should not be so long that the reflection is oversaturated, but should not be so short that the reflection is indistinguishable or undetectable. For example, when relying on Purkinje reflections, the pulse width should be short enough that the first Purkinje reflection is not oversaturated, but long enough that the fourth Purkinje reflection is detectable and identifiable. For example, 40 nanoseconds. Another implementation is to define two pulse widths that are more appropriate for each of the expected Purkinje image intensities. For example, 1 nanosecond pulses can be used for the first Purkinje reflection, and the pulse width can be increased to 40 nanoseconds to detect the fourth Purkinje reflection. In this case, the first Purkinje reflex can be detected as a larger dot due to the increased optical energy, while the fourth Purkinje reflex can be accurately detected. Furthermore, since the eye position ideally needs to be tracked in the 10 kHz range, a significantly lower duty cycle can be achieved. Using 1 nanosecond pulses and 100 samples for averaging the detected positions of the Purkinje reflexes, a duty cycle of (100 nanoseconds / 100 microseconds) = 0.1% can be achieved.

The system is also configured to calculate the lateral displacement and/or angular displacement μ, and/or the change thereof, based on the detection position of the reflected light on the optical sensor, or based on a change or displacement of the detection position. If this calculation is based on the detection position, the size of the optical sensor must be large. However, if it is based on the displacement of the detection position, the sensor can be much smaller. For example, after a change in the position of the eye is detected, the projection means can be realigned at the center of the optical sensor and then wait for another displacement of the detection position, which occurs periodically. For example, at a first eye position, the projection output is reflected and received at point A of the optical sensor, and at a second eye position after a certain time, the projection output is reflected and received at point B of the optical sensor. Based on the difference between points A and B, the system is configured or can calculate the lateral displacement and/or angular displacement of the eye.

The reflected light from the eye comprises at least one, preferably two, reflections, preferably at least a first Purkinje reflection, more preferably at least a first Purkinje reflection and a fourth Purkinje reflection. Based on the individual movements of the Purkinje reflections on the eye and on the optical sensor, and the relative movements between the reflections, the system can calculate the lateral and/or angular movements of the eye. An advantage of using the first and fourth Purkinje reflections is that both of them are located in the same optical plane as the pupil, since their images are located in the same optical plane as the pupil. However, the invention is generalizable and can be performed using other reflections, such as the second and third Purkinje reflections, or any combination of the four Purkinje reflections. Lateral translation, sometimes called translation, is the movement of the eye in the xyz directions, while angular translation, sometimes called pitch, yaw, and roll, is the rotation of the eye in all possible directions, for example.

For example, when the eyes are moved angularly (e.g., saccadic), this leads to a change in the position of the first and fourth Purkinje reflexes on the eye and on the optical sensor. Furthermore, as is known in the literature, angular movement of the eyes causes the first and fourth Purkinje reflexes to separate from each other (i.e., the separation between them changes, either increasing or decreasing their separation from each other). However, lateral movement of the eyes (i.e., movement within the eye socket) does not change the separation between the first and fourth Purkinje reflexes; instead, they move together by the same amount. Knowing this, and also knowing how the fourth Purkinje reflex moves relative to the first Purkinje reflex, it is possible to understand whether a purely lateral movement exists, or whether both an angular and a lateral movement exist. For example, when the gaze moves from one point to another, the angular displacement of the eye can be calculated by subtracting the change in the fourth Purkinje reflex (ΔP4) from the change in the first Purkinje reflex (ΔP1). This can enable analysis or a priori understanding of the readings of the optical sensor; for example, if the eye is displaced by A degrees, this will correspond to a displacement of B degrees between the fourth and first Purkinje reflexes. This is also advantageous in that the lateral displacement between the eye and the optical sensor is identified. This lateral displacement is not always lateral within the orbit; sometimes it can be a slippage of the system. For example, if the system moves relative to the eye, for example, if the system slides away from the user's face. Since the eye has a certain limit to how far it can move laterally, it is possible to understand whether the lateral displacement is due to slippage, a lateral movement of the eye, or a combination of these. Knowing the above, it is possible to distinguish between pure lateral movement, pure angular movement, and combinations thereof.

Unlike other systems, the pupil does not need to be tracked. Instead, the system relies on the first Purkinje reflex from the cornea and, optionally, the fourth Purkinje reflex from the lens of the eye. This is advantageous because pupil tracking is a complex process. Pupil tracking is inferior to tracking Purkinje reflexes because pupil images are typically distorted by the optical power of the cornea. Therefore, a corneal model is required to account for this optical power. However, when tracking the cornea, it relies entirely or nearly entirely on the first Purkinje reflex, and, optionally, on how it moves along the optical sensor.

The system is advantageous in that it utilizes a non-divergent projection means, such as a laser light source, which allows for an extended depth of focus within the user's eye. For example, it is possible to track the position of the eye even when it is not in optimal focus. This is evident because different users may require different focal points due to differences in the depth of their eyes relative to the system. For example, an image of the projection means is generated on the cornea and then reflected to an optical sensor.

While the present invention is described with reference to the first and fourth Purkinje reflexes, those skilled in the art will appreciate that other reflexes (such as the second and third Purkinje reflexes) and other combinations can also be used to achieve the present invention with varying degrees of convenience. Accordingly, the present invention is generalized to one or two reflexes that can provide information about the lateral and/or angular movements of the eye.

The system is configured such that the at least one reflection is received by the optical sensor. Specifically, the at least one projection means is configured or positioned such that the at least one reflection (e.g., a first Purkinje reflection, optionally together with a fourth Purkinje reflection) is received by the optical sensor. For example, the projection means is coupled to various lateral positions of the eye. For example, in case of movement of the eye, e.g., relative movement between the eye and the optical sensor, the projection means is moved accordingly to compensate so as to maintain the projection output at the same point on the cornea. Alternatively, the projection means itself need not be positioned or repositioned, but instead the projection output can be manipulated such that reflections are received by the optical sensor, e.g., using some mirrors or similar means.

There are three advantageous implementations discussed throughout the description. In a first implementation, the projection means is continuously repositioned so that the reflections remain on the cornea and are received by the optical sensor. In an alternative implementation, a reflecting means (e.g., a mirror or the like) is used to maintain the presence of the reflections. In another alternative implementation, multiple projection means are used, whereby each projection means corresponds to a different position of the eye, such that, for example, a projection means is used that causes reflections to be received by the optical sensor for each transverse position of the eye. Combinations of these implementations are also contemplated.

In the above system, the projection means (also the MEMS discussed below) is preferably positioned in the upper part, preferably on the left side (for the left eye) or on the right side (for the right eye) of the eye, since these areas in the user's field of view do not affect spatial and environmental perception.

Current-generation eye trackers, such as camera-based systems, are not accurate because they rely purely on certain user-directed actions to perform corrections, such as looking at specific stimuli in a specific order. Therefore, the best achievable accuracy is 0.5 degrees. The present system does not rely purely on user-directed actions. When combined with accurate SLAM, the present eye tracking system and method can achieve an accuracy better than 0.5 degrees. The system can also be used to determine user cognitive load and task saliency modality.

Because the eyes are tracked at very low latency and high speed, it may be possible to identify whether a person has certain medical conditions, such as Parkinson's disease or nystagmus, or other conditions that affect eye movement, and some of the symptoms thereof, as will be described later.

The system of the present invention is useful for simultaneous localization and mapping SLAM, for example, to understand where a user is looking. It may also be useful for optimizing interactive experiences in VR systems, for example, by allowing or assisting a user to select options through gaze. The present invention, in conjunction with SLAM, also enables precise gaze localization in real and virtual environments. The present invention also provides high accuracy, enabling optimization of foveation. The system of the present invention also enables gaze-contingent projection.

The above system is also advantageous in that it does not require post-processing of the data as is the case with prior art systems, for example, it does not require image processing of the data.

Preferably, said at least one projection means is one (but may be more than one) movable projection means. The movable projection means is positioned (e.g., angularly and transversely translated and rotated) such that said at least one reflection is received by said optical sensor. For example, to couple the projection means to different transverse and angular positions of the eye. Alternatively, and more preferably, said at least one projection means is a plurality of projection means, preferably arranged in a one-dimensional or two-dimensional array, for example, spaced laterally in a two-dimensional configuration. This is advantageous because it is easier, faster, and more accurate than the first scenarios where the projection means themselves are movable. For example, there is no need to move the projection means to acquire reflections, and for example, there is no need to move the light source. While a two-dimensional array scenario is preferred, a one-dimensional array scenario is also feasible. For example, it is possible to arrange a one-dimensional array along the horizontal axis of the eye, move the array along the vertical axis, or use mirrors to reflect the light in a vertical direction, for example within 10 to 30 degrees from the central line of sight.

The above-described plurality of projection means are preferably multiple VCSELs, which are advantageous because they provide single-mode operation and consume low power. VCSELs are also advantageous because they can be grouped into arrays and made addressable. VCSELs also have fast switching speeds. The output power for each VCSEL should be such that the light incident on the eye does not exceed 1.2 milliwatts in continuous wave (CW) operation, for eye safety reasons. The output power should be sufficiently strong to image the fourth Purkinje reflection, but not so strong that the first Purkinje reflection is oversaturated.

The VCSELs may have a wavelength in the near infrared range, preferably in the IR-A range, for example between 740 nm and 940 nm, for example 905 nm or 880 nm or 805 nm or 850 nm, or for example 1064 nm, or any other wavelength deemed suitable by one skilled in the art. The projection means are not limited to VCSELs and may be any other suitable type of light source. The optical sensor can read these wavelengths.

The system is configured to detect (e.g., identify and turn ON) at least one projector, among a plurality of projectors, that induces or allows at least a first Purkinje reflection, preferably a first Purkinje reflection and a fourth Purkinje reflection, to be received by the optical sensor. For example, the plurality of projectors may be addressable, such that the projectors may be selectively turned ON and OFF, for example, one or more may be turned ON simultaneously. For example, each projector in the array may be assigned an index, such that one projector has an index of 1, the projector next to it has an index of 2, and so on.

For example, the system is configured to detect, identify, and switch ON a projection means from which one or two Purkinje reflections are received by the optical sensor, preferably such that at least the first Purkinje reflection, more preferably the first and fourth Purkinje reflections, are received. Once this determination is made, the selected projection means is used during subsequent operations. For example, the projection means may be changed based on lateral movement of the eye or slippage of the system. This is advantageous because the system can be accommodated by simply changing the index, i.e., selecting a different projection means within the array, without the need for recalibration. The invention is also advantageous because only one or a small number of projection means are ON or switched ON at a time. Thus, only a portion of the eye is irradiated and imaged, allowing for power-efficient and fast operation. Preferably, at least one projection means among the plurality of projection means that induces at least the first Purkinje reflection is switched to another projection means based on the detection position and/or its displacement.

The output of each of the plurality of projectors may be collimated. Alternatively, the projectors may be imaged onto the eye by a microlens array disposed at the output of the plurality of projectors.

Having an array of projectors (e.g., VCSELs) is advantageous. For example, if the eye moves laterally, the system slips, or there is relative movement between the optical sensor and the eye, the system can continue to operate without moving the projectors. Instead, the system can switch to another VCSEL (e.g., an adjacent or near-adjacent pixel) that is optimal for the eye position. This is based on the displacement of the detection position of the reflected light on the optical sensor. For example, the minimum lateral movement that triggers switching to another VCSEL can be defined, or the minimum shift of the reflections that triggers switching of the VCSELs can be defined. Having an array of projectors also reduces system complexity. For example, switching to an adjacent VCSEL to receive the reflections is simpler than using a single, movable projector. In other words, different projectors are coupled to different lateral positions of the eye.

Another advantage is that it allows different users to use the same system. For example, it can be applied even when the second user's eye position differs from the first user's, for example, due to differences in interpupillary distance, resulting in lateral movement. In such cases, the operation can be switched to a different VCSEL, which may require recalibration. Therefore, the advantage of the present invention lies in its ability to implement a general-purpose system, rather than a customized one.

Ultimately, it is desirable to keep the reflection from the eye as close to or as close to the center of the optical sensor as possible. While the reflection will of course move around the sensor during operation, it is desirable to keep the reflection as close to the center of the sensor as possible, as this facilitates maintaining the widest possible tracking range. This can be achieved, for example, by selecting an appropriate VCSEL. For example, after slight angular and/or lateral movements of the eye that cause the reflection to shift away from the center of the optical sensor, the system can be adjusted (e.g., by moving the projection means or selecting a different projection means) so that the reflection returns to the center of the optical sensor or as close to it as possible. For example, the system can be configured to calculate how much the projection means must move for the reflection to return to the center of the optical sensor.

Having multiple projection means is also advantageous for achieving an extended depth of focus at the user's eyes. For example, this is advantageous when the eyes are positioned at different z-values or depths, for example, when the eyes are positioned deep in or protrude from the eye socket, or when the size of the nose causes the system to be closer or farther from the eyes. For example, suppose there is an optical system that focuses light onto the cornea. If User 1's eye is positioned at depth Z1, Projectionmeans 1 is appropriate. However, if User 2's eye is positioned at depth Z2, and Z2 is closer or farther from Z1 by multiple projection means, then Projectionmeans 2 is more appropriate for operation. In other words, the optimal projection means for User 1's eye position and the optimal projection means for User 2's eye position are different. In other words, instead of moving the system itself closer or farther from the user to compensate for the difference in the z-direction positions of the eyes, different projection means are selected and used.

Preferably, the optical sensor is gated, meaning that it only responds to photons observed within a short time period. This is advantageous for reducing noise caused by ambient light. This is especially important in bright environments where a lot of ambient light is expected. For example, the optical sensor is configured to be synchronized with the projection device, so that its output is read only during the time the projection device is operating.

Preferably, the optical sensor comprises a plurality of sensing units forming a matrix of a plurality of pixels. Each sensing unit comprises a photodetector. The system is configured to calculate a displacement of a detection position based on a displacement of reflected light on the sensing unit. For example, if a first Purkinje reflection is displaced by X pixels, a corresponding angular displacement (e.g., a saccade) can be calculated.

Preferably, the optical sensor has a field of view (FOV) of at least 5 millimeters on the image plane along the horizontal axis of the eye (e.g., the direction connecting the nose and the ear), preferably at least 10 millimeters, and more preferably at least 15 millimeters. For example, 5 millimeters corresponds to a field of view of about 30 degrees (horizontal). A realistic field of view of the eye is known to be about 32 degrees (horizontal) and about 24 degrees (vertical), for example, a cone of view of about 30 degrees. This means that the minimum FOV of the optical sensor on the image plane is about 5.3 millimeters (horizontal) and about 4.2 millimeters (vertical). However, to account for the interpupillary distance of the human population, a preferred minimum FOV is about 15 millimeters in the horizontal direction, and the vertical direction is kept to about 4 to 5 millimeters, depending on the use case.

For example, the optical sensor can image an area of at least 5 millimeters by 5 millimeters, preferably at least 10 millimeters by 10 millimeters, more preferably at least 15 millimeters by 15 millimeters, configured to encompass at least 5 millimeters, preferably at least 10 millimeters, more preferably at least 15 millimeters along a horizontal axis of the eye on the image plane, the horizontal direction being as described above.

The field of view (FOV) can be covered by a single optical sensor, for example, a stationary optical sensor, for example, an optical sensor having a large field of view, for example, an optical sensor having a field of view of at least 5 millimeters, at least 10 millimeters, or at least 15 millimeters or more. Alternatively, the field of view can be covered by a single movable optical sensor, for example, an optical sensor having a small field of view, for example, 5 millimeters, but being able to move so as to cover a large field of view overall, for example, at least 10 millimeters, or at least 15 millimeters or more. For example, the optical sensor can be movable, thereby allowing the optical sensor to have an overall field of view of at least 10 millimeters, or at least 15 millimeters or more. Alternatively, the field of view can be covered by a plurality of optical sensors, for example, each having a small field of view of a few millimeters. The optical sensors are arranged such that the overall field of view of the plurality of optical sensors is at least 5 millimeters, at least 10 millimeters, or at least 15 millimeters or more.

In other words, the field of view in the image plane should preferably include the area where angular translation of the eyes can occur. This has the advantage of making the system suitable for a variety of users with different interpupillary distances. The field of view required to detect angular eye movement (e.g., at all extreme orientations) is approximately 5 millimeters, and since people's interpupillary distance can vary by approximately 10 millimeters, for example, the eyes may be positioned closer or farther from the nose and ears. Therefore, the system can track eyes within a wide field of view, making it suitable for almost all users. Therefore, the most desirable scenario is a total field of view of approximately 15 millimeters. The size of the optical sensor should be appropriately calculated depending on the imaging ratio. For example, for a 1:5 image magnification, the optical sensor size would be approximately 3 millimeters, which is a reasonable size.

Similarly, in the case of multiple projection means, the multiple projection means are configured to cover the same area as described above. For example, the VCSEL array covers at least 5 millimeters, preferably at least 10 millimeters, and more preferably at least 15 millimeters, as described above. Therefore, different VCSELs are configured to provide optimal operation for different users.

Preferably, the projection means is configured to be re-orientable, so that the projection output remains at the determined portion or region of the cornea even when the eye rotates. This is to maintain the first Purkinje reflection at a specific location on the optical sensor. For example, the projection means always follows the eye, and the projection output always reaches the cornea. This is based on the detection position and/or displacement of the detection position in the optical sensor. In other words, the optical sensor drives the projection means so that the projection means and thus the projection output follows the cornea. For example, the optical sensor immediately informs the projection means of the displacement of the detection position (e.g., using a feedback loop) and, for example, informs the projection means of the approximate amount and direction of the eye rotation, at which point the projection means should be re-oriented to follow the cornea and reach the projection output on the cornea. For example, the projection output always remains within a certain range of the cornea. This has the advantage of allowing the eye to be continuously tracked. For example, one could use the first Purkinje reflex, which provides information about at least angular eye movements (e.g., saccades), or, additionally, the fourth Purkinje reflex, which provides information about lateral eye movements. The method could also achieve the same result by manipulating the output using a mirror or reflector, rather than reorienting the projection means.

Preferably, the system also includes a reflecting means, preferably a dynamic deflection mirror, more preferably a MEMS. The projection output is projected onto the reflecting means such that the reflecting means reflects the output onto the determined area or portion of the cornea. The reflecting means is configured to be repositionable so that the projection output remains on the determined area of the cornea even when the eye rotates. For example, there is a single MEMS capable of reflecting light from the projection means or multiple projection means. Reorientation is performed based on a displacement of the detection position or positions in the optical sensor. In other words, the projection means can be repositioned to maintain the projection output along the cornea (as described above), or the MEMS can be configured to receive the projection output and rotate along the cornea. The use of MEMS has the advantages of fast response time, low power consumption, and that it is easier to rotate the MEMS than to move the projection means. For example, reflecting means such as MEMS do not consume power when stationary. The reflective means or MEMS in the present invention have two main purposes: compensation, which will be discussed later, and maintaining light incident on the cornea.

The angle of incidence of the incident light (beam) relative to the cornea remains the same or substantially the same, and the area of the light reaching the cornea also remains the same or substantially the same. For example, the angle of incidence can be between 10 and 40 degrees, measured from the axis defining the central gaze of the eye. The angle of incidence needs to be optimized so that, if more than one reflection (e.g., the first and fourth Purkinje reflections) are present, they do not overlap or appear in the same area on the optical sensor. If they overlap, the reflections will be indistinguishable.

The angle of incidence corresponds to the angular range over which the MEMS rotates, and should preferably be minimized. This can be achieved by having multiple projection means or VCSELs, so that if the eye moves laterally (e.g. the system slides off the user's face), a different VCSEL can be used instead of rotating the MEMS by an extreme angle. This ensures that the change in the first Purkinje reflection is purely or almost purely due to the angular translation of the eye, resulting in improved eye tracking performance.

The angle at which light enters the eye depends on the system design. For example, the projector emits light at a certain angle below which the reflection can be received by the optical sensor. Furthermore, the reflector may be aligned with the center of rotation of the eye so that the beam strikes the cornea and reaches the same point. In other words, the projector or light source forms an image on the cornea.

The combination of a projection means having a low numerical aperture beam and a reflective means (e.g., a MEMS) is advantageous because the MEMS can be coupled to the rotational center of the eye without having to use a large beam covering a large area or constantly moving the projection means to correspond to the new rotational center of the eye. The displacement of the optical sensor and the reflection position on the optical sensor drives the rotation of the MEMS. That is, the MEMS is in a conjugate plane relationship to the rotational center of the eye, and as the eye rotates, the MEMS rotates accordingly. For example, the optical sensor can be coupled to the reflection means via a feedback loop. Alternatively, in the absence of MEMS, the projection means can be coupled to the rotational center of the eye, so that as the eye rotates, the projection means rotates as well.

For example, a reflective means (e.g., a MEMS) is reoriented based on a displacement of the detection location, and the beam is maintained so that it reaches substantially the same point on the cornea within a predetermined tolerance. This can be done continuously and rapidly, so that the beam always follows the cornea, for example, so that the cornea remains within a certain range of the projection output.

That is, one of the main purposes of the MEMS is to maintain the presence of the Purkinje reflection. For example, the MEMS rotates to maintain illumination on the cornea. If the eye's gaze changes so that the first Purkinje reflection moves away from the center of the optical sensor, it is possible to determine whether the movement is angular, lateral, or a combination of the two based on the movement of the first Purkinje reflection and possibly the fourth Purkinje reflection. Accordingly, the MEMS rotates so as to maintain the presence of the Purkinje reflection on the optical sensor, preferably at the center of the sensor. However, if the eye movement is purely lateral, as described above, the Purkinje reflection can be returned to the center of the sensor by switching to a different VCSEL source without moving the MEMS.

It is desirable that the rotation of the MEMS be based on changes in the detection position, rather than the absolute value. For example, if there is a corresponding change in the rotation of the MEMS when the detection position changes, the reflected light will remain centered on the optical sensor. Simply rotating the MEMS slightly will not cause a change in the position of the Purkinje reflex; only angular or lateral movement of the eye will cause a change in position.

The selected MEMS must be capable of operating in both open-loop and closed-loop configurations. For example, the MEMS can operate in open-loop, quasistatic or resonant mode, or in closed-loop mode when an optical sensor drives the MEMS operation. Furthermore, the MEMS can be operated in open-loop mode during calibration.

The feedback loop is based on the first Purkinje reflex, which provides information about the lateral and angular displacement of the eye. However, the fourth Purkinje reflex is also useful for understanding the lateral displacement of the eye, for example, when switching between different VCSELs. For example, if the gaze moves from one point to another, the net angular displacement of the eye can be calculated by subtracting the change in the fourth Purkinje reflex (ΔP4) from the change in the first Purkinje reflex (ΔP1).

This invention has the advantage of enabling low-latency eye tracking, which reduces the sensor's size requirements. That is, since the sensor detects changes in the detection position and adjusts the MEMS and/or VCSEL array accordingly to redirect the reflected light back toward the optical sensor's center, a very large sensor is not required.

Preferably, the size of the optics within the system is minimized, thereby minimizing the overall size of the system. This can be achieved, for example, by minimizing the movement of reflective means, such as MEMS, relative to changes in the angular movement of the eye. For example, the ratio between the angular movement of the eye and the required movement of the MEMS can be defined as a scaling factor, and this scaling factor is optimized to minimize the movement of the MEMS. That is, even if the angular movement of the eye is large, the MEMS moves only as much as necessary, and, for example, does not move excessively unnecessarily, thereby minimizing the size of the optics. In other words, the larger the MEMS movement, the larger the required optics size.

Preferably, the output signal of the optical sensor has a repetition rate 5 to 10 times higher than the resonant frequency of the MEMS. For example, the repetition rate of the output signal is at least 1 kHz, and preferably at least 10 kHz or higher. This allows the MEMS to rotate rapidly to track the eye based on the optical sensor output signal, thereby improving accuracy and reducing noise.

Preferably, the projection output of the projection means impinges on one side or the center of the cornea. One side of the cornea is preferably irradiated, as the Purkinje reflex typically appears on the side of the cornea. However, the present invention is generalized to irradiate any area of the cornea, and the Purkinje reflex can still be obtained in this case. This is a design choice and depends on the location of other components within the system.

Preferably, the system is configured to find and turn on at least one, or two, or three or more, of the multiple projection means. For example, by turning on one or more projection means at a time, the system searches for, selects, and turns on a projection means that can receive the first Purkinje reflex, and particularly the weaker fourth Purkinje reflex, by the optical sensor. For example, in addition to the projection means that can receive the first Purkinje reflex, one or more other projection means that can receive the fourth Purkinje reflex may be used. This is advantageous in cases where the pupil size is small, for example, when the user is in bright lighting conditions, and then more than one projection means may be required to allow the optical sensor to receive the fourth Purkinje reflex, which is generally a weaker reflection. For example, in the case of a small pupil, the projection means required to generate the fourth Purkinje reflex may not be the optimal projection means described above, and may require, for example, a second or third projection means, i.e., a projection means that is offset from the optimal projection means. This is another advantage of having multiple projection means in an array configuration.

Preferably, to initiate motion, the system is configured to calibrate by scanning the projection output of the projection means along a predetermined scanning path on the cornea. For example, this is performed to initiate motion. During calibration, the most suitable projection means among multiple projection means is found and selected. This is performed in the following manner. First, multiple projection means are turned on sequentially, for example, one by one. At each point in time and for each projection means, an optical sensor is observed. The projection means that produces a Purkinje reflex is selected. Typically, this will be a group of adjacent projection means, and may also include some outlier projection means, for example, due to noise. The reflection means (e.g., MEMS) then scans each resulting projection means along a known scanning path (e.g., Lissajous scan, raster scan, or other suitable scan). The projection means that produces the least movement of the Purkinje reflex during the scan is selected, and thus becomes the projection means used in the eye tracking process. For example, the light source with the least lateral motion of the P1 and P4 reflections is selected. This is because the most suitable projection medium for the operation has the least decentering from the optical axis of the cornea. In other words, for the most suitable projection medium, scanning the MEMS along a known scanning path, for example, within a small area, within the corneal range, does not cause abrupt shift of the Purkinje reflection, and its output is still projected onto the cornea. For example, after scanning or rotating the MEMS by a certain amount, the projection medium with the correct (e.g., expected) reflection ratio between P1 and P4 is selected. For example, this can be done by applying formula (1) of (High-resolution eye-tracking via digital imaging of Purkinje reflections, Ruei-Ju Wu et al., https://doi.org/10.1101/2022.08.16.504076).

Preferably, each of the photodetectors is a single photon detector, preferably a single photon avalanche detector (SPAD), capable of detecting a single photon. SPADs are useful because they have a low response time with a minimum photon budget. Because SPADs can detect single photons, they are highly accurate and have a high dynamic range. This is advantageous in that they can capture the fourth Purkinje reflection, which is a weak reflection. Because SPADs are sensitive to single photons, they can also detect weak signals. For example, SPADs are advantageous in that they can detect both the first and fourth reflections, which have an intensity difference of three orders of magnitude. The low response time of the photodetectors is also advantageous, because, for example, photons are detected in nanoseconds and encoded into digital signals, so that the eye is continuously tracked and the projection output always remains within a certain range from a predetermined area of the cornea. Low response time (or low latency) is also advantageous for quickly detecting position changes in a photodetector and subsequently transmitting this change information to a reflective means or MEMS, for example. SPADs are also useful because of their high photon sensitivity, which means that their projection output requires minimal energy.

Preferably, the at least one photosensor comprises a first photosensor and a second photosensor, each of which is configured to receive a different reflection. For example, the first photosensor is configured to receive the first Purkinje reflex, and the second photosensor is configured to receive the fourth Purkinje reflex. For example, the second photosensor may be configured to be more sensitive to detect the fourth Purkinje reflex (a weaker reflection). For example, the light entering the photosensor may be split into two parts, with each photosensor receiving one of the parts. This is advantageous in bright light environments, because the pupil is small in this case, making the fourth Purkinje reflex very weak.

Preferably, the system is configured to locate two projection means, each of which is configured to generate a different Purkinje reflex. For example, the system may be configured such that one or more projection means generate a first Purkinje reflex, and one or more other projection means generate a fourth Purkinje reflex.

Preferably, the light sensor is positioned in the conjugate plane of the pupil. This is the plane where the hypothetical first Purkinje reflection and the actual fourth Purkinje reflection are focused. This is advantageous in that there is a clear ratio between the angular displacement of the eye and the displacement observed by the light sensor. For example, if the eye is angularly displaced by A degrees, this corresponds to a detected displacement of B degrees by the light sensor, where A and B are related by a known ratio, which is a calibration factor that varies from user to user. For example, the calibration factor is determined during the initial calibration of the system and varies from user to user. For example, the user can be asked to rotate their eyes along a specific pattern, and the calibration factor can be derived based on how much the eye has moved and how much the sensor has shifted as the reflection changes accordingly. For example, this can be determined by calculating which angular and/or lateral displacements result in a certain displacement observed by the light sensor. The calibration factor is user-specific and varies depending on the corneal characteristics of each user. The above calibration factor is not the same for each individual because it is affected by other characteristics of the eye, such as optical aberration and the corresponding cylinder distortion in the photosensor. This cylinder distortion is not the same in the vertical and horizontal directions. Furthermore, for some individuals, distortion may exist in both the vertical and horizontal directions. Therefore, if the eye moves in both the horizontal and vertical directions, cylinder distortion in both directions will affect the eye. Therefore, the calibration factor is not the same in the horizontal and vertical directions. The system is advantageous in that it is configured to detect distortion in both directions. This is done during the calibration process. For example, the system asks the user to move their eyes horizontally, vertically, or in a combination of the two directions, and then calculates the amount of this distortion. In many cases, this distortion is a combination of vertical and horizontal distortion, so it is desirable to understand the displacement observed in the photosensor when the eye moves in a combination of vertical and horizontal directions.

Preferably, the projection means is operated in a pulsed mode. For example, each pulse is tuned to produce a different Purkinje reflection in the light sensor. For example, one pulse of weak or normal intensity produces the first Purkinje reflection, which is a strong reflection, and one pulse of strong intensity produces the fourth Purkinje reflection, which is a weak reflection and is therefore necessary. Alternatively, these pulses may have different lengths. For example, there may be a short pulse for observing the first Purkinje reflection and a long pulse for observing the fourth Purkinje reflection. Alternatively, the light sensor may be configured to have two regions of interest, one region expected to receive the first Purkinje reflection and the other region expected to receive the fourth Purkinje reflection. For example, one region of interest may have one sensitivity and the other region of interest may have a different sensitivity, thereby capturing both the first and fourth Purkinje reflections. To achieve this, during the calibration phase, it is necessary to estimate where on the light sensor the first strong Purkinje reflection is received and where the fourth weak reflection is received. For example, if the detection location size on the light sensor is large, the detection is more likely to be the first Purkinje reflection, and similarly, if the detection location size on the light sensor is small, the detection is more likely to be the fourth Purkinje reflection. Then, different regions of interest may include pixels or SPADs with different bias values or different observation windows, for example, to suit the first or fourth Purkinje reflection. Then, during calibration, that is, normal operation, the system tracks the relative positions of the reflections. Alternatively, the pixel sensors may have tunable sensitivities. For example, two measurements may be taken, one with one sensitivity and the other with a different sensitivity. For example, by adjusting the sensitivity of the light sensor, sometimes the first Purkinje reflection may be detected and sometimes the fourth Purkinje reflection may be detected. The above solutions are advantageous in addressing the problem that the first and fourth Purkinje reflexes differ in intensity by three orders of magnitude. Another way to address this problem is to filter the data from the photosensor from noise, such as ambient light, to reduce the number of falsely detected sensing units, thus making it easier to identify the fourth weak Purkinje reflex. For example, detections from adjacent sensing units (pixels) can be checked, as Purkinje reflexes are typically detected in more than one pixel (see PCT EP2021 087594). Alternatively, or additionally, detections can be checked to see if they persist over time (see PCT IB2022 058609), as the eye does not move dramatically over very short periods of time, and thus the Purkinje reflex does not move dramatically over very short periods of time. For example, the eye does not move dramatically over 40 nanoseconds, or even 100 nanoseconds. Therefore, detections that persist for a long period of time are likely true detections, while those that do not are likely false detections. Alternatively or additionally, each detector may be connected to a single column bus and a single row bus, as described in PCT/IB2021/054688, to avoid reading the output pixels pixel by pixel, which is advantageous for reducing noise and making the system faster. Other filtering mechanisms may be considered, as described in PCT IB2021 054688, PCT EP2021 087594, PCT IB 2022 000323, and PCT IB2022 058609.

For example, the system is adapted to filter the detection locations determined by the plurality of sensing units, e.g., to separate reflected light from ambient light and noise. This may be based on whether the detection exists in one or more adjacent or semi-adjacent sensing units, e.g., only considering a pixel in a true state if at least one adjacent pixel is also in a true state. This may additionally or alternatively be based on whether the detection exists in at least two consecutive time steps, e.g., only considering a pixel in a true state among a plurality of pixels obtained in an observation if at least one adjacent pixel is also in a true state, or based on a combination of at least two consecutive or semi-consecutive observation windows.

For example, where each pulse has a first predetermined intensity or a second predetermined intensity, or where each pulse has a first predetermined pulse width or a second predetermined pulse width. The first predetermined intensity or the first predetermined pulse width is adapted to cause one reflection (e.g., a first Purkinje reflection) to be received by the optical sensor, and the second predetermined intensity or the second predetermined pulse width is adapted to cause another reflection (e.g., a fourth Purkinje reflection) to be received by the optical sensor. For example, the first predetermined intensity is a low intensity at which a first strong Purkinje reflection can be received by the optical sensor, and the second predetermined intensity is a high intensity at which a fourth weak Purkinje reflection can be received by the optical sensor. Similarly, for example, the first predetermined pulse width is shorter than the second predetermined width because the first Purkinje reflection is stronger than the fourth Purkinje reflection.

Preferably, the optical sensor is configured to measure the position of the reflection and output the position as an output of the optical sensor. In other words, the optical sensor outputs only the detection position, i.e., the sensing unit that detected the detection, thereby eliminating the need to read each of the multiple sensing units individually. This is described in PCT/IB2021/054688. This has the advantage of saving energy and reading time of the sensing unit by reading only the units that have detected the detection. In addition, as a specific implementation, the detection area can be calculated and the average position of the detection can be determined as a result, thereby indicating the average position at which the Purkinje reflex falls.

Preferably, the projection means is operated in a pulsed mode having a known pulsed frequency. The system is adapted to minimize or eliminate unwanted signals and noise resulting from external light sources illuminating the eye, etc. This is achieved by considering only detections having the known pulsed frequency of the projection means or substantially the same frequency, and discarding detections having different or substantially different pulsed frequencies.

Preferably, the optical sensor is configured to simultaneously measure at least two, preferably two, detection positions on the optical sensor and to provide said positions at the output of the optical sensor. Preferably, each position corresponds to a different specular reflection. Advantageously, one detection position corresponds to the first Purkinje reflex (P1), and the other detection position Δ corresponds to the fourth Purkinje reflex (P4). This has the advantage of allowing calculation of lateral and/or angular movements of the eye that are partially or fully dependent on both the first and fourth Purkinje reflexes.

In a second aspect, the present invention relates to a display system using the eye tracking system according to the first aspect. For example, a VR system that displays a scene according to the direction of gaze.

In a third aspect, the present invention relates to an eye tracking method, for example, corresponding to the system of the first aspect. The method comprises the step of projecting a projection output onto a predetermined area of the cornea of the eye by at least one projection means. The method also comprises the step of detecting a reflection from the eye by at least one optical sensor. The reflection from the eye comprises at least one reflection, preferably two reflections. The method also comprises the step of determining a detection position of the reflection on the optical sensor. The method also comprises the step of calculating a direction and/or position of the eye, for example, a lateral and/or angular movement and/or a change thereof, based on the detection position and/or the displacement thereof.

Preferably, the method comprises configuring the projection means to be movable. For example, the method comprises a step of moving or positioning the projection means such that at least one reflection is received by the optical sensor. Alternatively, the method comprises a step of configuring the projection means as a plurality of projection means arranged in a one-dimensional or two-dimensional array. The method further comprises a step of locating and selecting at least one projection means and switching it such that at least one reflection, preferably at least a first Purkinje reflection and a fourth Purkinje reflection, is received by the optical sensor.

Preferably, the method comprises a step of switching the operation from one projection means to another based on the detection position and/or displacement thereof, for example, based on a lateral movement of the eye. For example, if the first projection means no longer produces a reflection from the eye, this may be due to a lateral displacement of the eye or a slippage of the device. In general, it is preferred to power on only one projection means at a time, or a minimum number of projection means as needed (e.g., one for the first Purkinje reflex and one for the fourth Purkinje reflex). For example, unnecessary redundant light sources are turned off. However, if multiple groups of projection means are needed at a time to increase the intensity of the reflection or to induce different reflections, the method comprises a step of switching the operation from one group of projection means to another group of projection means.

Preferably, the method comprises configuring an optical sensor to include a plurality of sensing units forming a pixel matrix, each of which includes a photodetector.

Preferably, the method comprises configuring at least one optical sensor to have a minimum field of view of at least 5 millimeters along a horizontal axis of the eye on the eye image plane, preferably at least 10 millimeters, more preferably at least 15 millimeters. The method further comprises covering the field of view with a single optical sensor having at least the minimum field of view. Alternatively, the method comprises covering the field of view with a single movable optical sensor that can move its position so that the entire field of view is at least the minimum field of view. Alternatively, the method comprises covering the minimum field of view with a plurality of optical sensors so that the entire field of view is at least the minimum field of view.

Preferably, the method comprises a step of reorienting the projection means so that the projection output remains in the predetermined corneal region when the eye rotates. The reorientation is based on the detection position and/or its displacement.

Preferably, the method comprises a step of reflecting the projection output onto the predetermined corneal area by a reflecting means. For example, after the projection output is projected onto the reflecting means, the output is reflected onto the predetermined corneal area. The method also comprises a step of reorienting the reflecting means so that the projection output remains on the predetermined corneal area when the eye rotates. The reorientation is based on the detection position and/or its displacement.

Preferably, the method comprises the step of finding at least one other projection means among the plurality of projection means that causes the optical sensor to receive a fourth Purkinje reflection.

Preferably, the method comprises a step of calibrating the system. The calibration comprises a step of scanning the projection output of the projection means along a predetermined scanning path on the cornea.

Preferably, the method comprises the step of providing each of the photodetectors with a single photon avalanche detector (SPAD), wherein the SPAD is capable of detecting a single photon.

Preferably, the method comprises providing a first optical sensor and a second optical sensor. The method also comprises configuring the first optical sensor to receive one reflection and configuring the second optical sensor to receive the other reflection. This is advantageous when the intensities of the reflections are different.

Preferably, the method comprises the step of placing the optical sensor on the standard plane of the pupil.

Preferably, the method comprises a step of filtering data of the sensing units, for example, based on whether the detection occurs in one or more adjacent or semi-adjacent sensing units, and/or based on whether the detection occurs in at least two consecutive time steps.

Additional features and advantages of embodiments of the present invention will be described with reference to the drawings. It should be noted that the invention is not limited to the specific embodiments illustrated or described in the drawings, but is limited only by the scope of the claims.

Figure 1 schematically illustrates an eye tracking system (100) according to the present invention. In Figure 1, a projector (3) projects an output (4) onto the cornea of an eye (1), preferably a predetermined portion (2) of the cornea, preferably one side of the cornea, for example the left or right. The incident light (4) is then reflected (5) onto an optical sensor (6). The optical sensor (6) comprises a plurality of pixels (7) arranged in a matrix. Based on the reflected light (5) received by the optical sensor (6), this indicates an angular or lateral displacement of the eye, and the direction of the projector is changed to follow the cornea.

Figure 2 shows different layers (41-44) and different Purkinje reflexes (P1-P4) of the eye (1) according to the present invention. The most important layers are the cornea (41) and the lens (44). The light reflected from the cornea (41) is called the first Purkinje reflex (P1), and the light reflected from the lens (44) is called the fourth Purkinje reflex (P4).

Figure 3 schematically illustrates an alternative configuration of an eye tracking system (100). In this case, a reflecting means or MEMS (10) receives the projection output (4) and reflects it onto the cornea. Using a feedback loop (11), an optical sensor (6) drives the movement of the MEMS (10) along with each displacement of the eye (1) (e.g., rapid eye movement, saccade) based on the displacement of the Purkinje reflex (P1, P4) on the optical sensor (6).

Figure 4 schematically illustrates an alternative configuration of an eye tracking system (100). The difference from Figure 3 is the presence of multiple projectors (3). During the calibration phase, the projectors are turned on one at a time to find a projector that generates the first Purkinje reflection and, if necessary, the fourth Purkinje reflection. This results in finding a group of projectors that satisfy these conditions. The optimal projector is then selected from this group. For example, the projectors can be turned on one at a time and scanned using a known pattern, such as a Lissajous pattern, to determine how the first reflection and, if necessary, the fourth reflection, move across the optical sensor. This allows the system to be calibrated and the projector most suitable for system operation. This calibration only needs to be performed once. Having multiple projectors is useful for adjusting to different lateral positions of the eye. For example, if the eye moves laterally or the system slides off the user's face, switching to a different projector is more efficient and faster than moving the MEMS.

Figure 5 shows a light source or emitter (13), the output of which falls on a mirror (10). Depending on the orientation of the mirror (10), the light is reflected at a certain angle. The light passes through some optical system that focuses the light on the cornea (2) of the eye (1). For example, the eye (1) moves from position (a) to another position (b). By properly rotating the MEMS, the light remains at the same point, and thus the eye is continuously tracked. Those skilled in the art will appreciate that the system can be arranged in various ways, and for example, there may be many other alternatives for the optical system used in the system, so that the system is not limited by this implementation. As can be seen from the reference numeral (20), the mirror (10) and the eye (1) are conjugate planes.

Figure 6 shows an exemplary implementation of the system. In (a), three VCSELs are arranged in an array, and there is also an optical system that focuses light onto the eye (1). By switching between the VCSELs, the operation can be switched to different lateral positions of the eye. For example, the image of VCSEL (14) is generated in area (19), similarly (15) is generated in area (18), and (16) is generated in area (17). This allows for simply switching to a different VCSEL, rather than rotating the MEMS to a high or extreme angle. In other words, the MEMS is designed to be rotated only within a small angular range, for example, 5, 10, or 15 degrees measured from the central line of sight. Consequently, the change in Purkinje reflection is mostly due to angular movement of the eye, with a smaller effect due to lateral movement. This enables accurate eye tracking and facilitates convergence of the closed-loop operation.

This system can be further optimized by changing the illumination direction of the VCSELs in (b). For example, by ensuring that the VCSEL output intersects the optics or lens. In (c), the system can be further optimized by placing the optics closer to the eye, ensuring that the VCSEL output intersects before the optics or lens, and placing a MEMS mirror at the point where the VCSEL outputs intersect. Of course, the optics or lens used determines the magnification of this part, and thus the size of the VCSEL array.

Figure 7 shows another implementation of the system, which includes a VCSEL array (14-16). The VCSELs are selected based on their lateral position (17-19) relative to the eye. In this implementation, a reflective surface (21) reflects light into the eye with the help of a rotatable mirror (10).

Figure 8 (a) shows an implementation similar to Figure 7, considering only one lateral position of the eye and a single light source. In this case, a pinhole (23) is present, but is not always required. A second mirror (22) is also present. Figure 8 (b) shows that the eye can be tracked at different angular positions by adjusting or rotating the mirror (10).

Figure 9 shows an implementation of the system, showing two VCSELs (13-14). As discussed above, coupling to different lateral positions of the eye can be achieved by switching between the VCSELs. However, another challenge is coupling to different z-positions of the eye. This requires rotating the MEMS while simultaneously switching the VCSELs. For example, a Kalman filter can be used to calculate the required lateral and angular changes, for example, which VCSEL to switch to and how much to rotate the mirror. For example, using the first VCSEL (14) in the first position and the first rotation of the MEMS (dotted MEMS) produces an image at the first z-position (open circle). Conversely, using the second VCSEL (13) in the second position and the second rotation of the MEMS (solid MEMS) produces an image at the second z-position (solid circle).

Other configurations for achieving the objectives of the methods and devices for implementing the invention will be apparent to those skilled in the art. The following description provides details of specific embodiments of the invention. However, no matter how detailed the description may appear in the text, it should be clear that the invention can be applied in various ways. It should also be noted that the use of specific terms when describing specific features or aspects of the invention should not be construed as limiting the definition to the specific features or aspects to which those terms are combined.

1: Eyes
2: Cornea or a predetermined area of the cornea
3: Projection means or laser light source
4: Projection output or laser beam
5: Reflected light
6: Optical sensor
7: Pixel
8: Photodetector
9: Ambient light
10: Reflective means or MEMS
11: Feedback loop
12: Optics
13: Emitter
14, 15, 16: Light sources or VCSELs
17, 18, 19: Different transverse positions of the eyes
20: Conjugate planes
21: Reflective surface
22: Mirror
23: Pinhole
100: Eye Tracking System
41: Cornea
42: Anterior chamber
43: Iris
44: Lens
P1: Purkinje reflex 1
P2: Second Purkinje reflex
P3: Third Purkinje reflex
P4: Fourth Purkinje reflex

Claims (14)

As a method of tracking eye orientation:
a. A step of illuminating the cornea (41) of the eye (1) tracked by a light beam (4) on a predetermined area (2) of the cornea (41), preferably in a pulsing mode, by means of a projection means (3) having a substantially small numerical aperture;
b. A step of detecting light (5) reflected from the eye (1) by at least one optical sensor (6), wherein the reflected light (5) comprises at least one, preferably two, specular reflections, the reflections preferably comprising at least a first Purkinje reflection (P1), more preferably at least a first Purkinje reflection (P1) and a fourth Purkinje reflection (P4);
c. A step of determining a detection position on at least one optical sensor (6), preferably wherein the optical sensor (6) comprises a plurality of sensing units (7) forming a matrix of pixels, preferably wherein each sensing unit (7) comprises a photodetector (8), preferably wherein each photodetector is a single photon avalanche detector (SPAD);
d. A step of calculating the direction and/or position and/or change thereof of the eye (1) based on the determined detection position and/or displacement thereof;
Including,
- if the projection means (3) is a single movable projection means, the method comprises a step of positioning the projection means (3) so that the reflected light (5) is received by the optical sensor (6); or
- A method, wherein when the projection means (3) is a plurality of projection means arranged in a one-dimensional or two-dimensional array, the method includes a step of finding at least one projection means that causes the reflected light (5) to be received by the optical sensor (6). In claim 1, the method further comprises a step of switching the operation of the plurality of projection means from one projection means to another projection means based on the detection position and/or displacement thereof, such that the reflected light (5) is received by the optical sensor (6). A method according to claim 1 or 2, further comprising a step of filtering a detection location determined by one of the plurality of sensing units (7) based on the presence or absence of the detection above at least one sensing unit adjacent to the one sensing unit, or based on the presence or absence of the detection in at least two consecutive time steps. In any one of claims 1 to 3, the method further comprises the step of configuring the at least one optical sensor to have a minimum field of view of at least 5 millimeters, preferably at least 10 millimeters, more preferably at least 15 millimeters, in the eye image plane along the horizontal axis of the eye (1),
The above view is:
- one optical sensor (6) having a total field of view of at least the minimum field of view, or
- one movable optical sensor (6) capable of being moved to a position that provides a total field of view of at least the minimum field of view, or
- Multiple optical sensors (6) having a total field of view at least greater than the minimum field of view
covered by, method. A method according to any one of claims 1 to 4, wherein the method further comprises a step of reorienting the projection means (3) so that the light beam (4) remains in a predetermined area of the cornea (2) when the eye (1) rotates, wherein the reorientation is based on the detection position and/or its displacement. In any one of claims 1 to 5, the method comprises:
- A step of projecting the above light beam (4) onto a reflecting means (10);
- A step of reflecting the light beam (4) into a predetermined area of the cornea (2);
- A step of reorienting the reflecting means (10) so that the light beam (4) is maintained in a predetermined area of the cornea (2) when the eye (1) rotates based on the detection position and/or its displacement.
A method further comprising: A method according to any one of claims 1 to 6, wherein the method further comprises the step of finding at least one other projection means (3) among the plurality of projection means (3) that causes the fourth Purkinje reflection (P4) to be received by the optical sensor (6). A method according to any one of claims 1 to 7, wherein the method further comprises a step of scanning the light beam (4) along a predetermined scanning trajectory on the cornea (2). In any one of claims 1 to 8, the at least one optical sensor (6) comprises a first optical sensor and a second optical sensor, and the method comprises:
- a step of configuring the first optical sensor to receive one reflection; and
- a step of configuring the second optical sensor to receive one other reflection;
A method comprising: A method according to any one of claims 1 to 9, wherein the method comprises the step of positioning the optical sensor (6) on a conjugate plane of the pupil (2). In any one of claims 1 to 10, the method comprises:
- A step of operating the above projection means (3) in a pulsed mode;
- Steps to configure each pulse to have:
■ First predetermined intensity or second predetermined intensity, or
■ First predetermined pulse width or second predetermined pulse width;
- a step of adjusting the first predetermined intensity or the first predetermined pulse width so that one reflection is received by the optical sensor (6); and
- a step of adjusting the second predetermined intensity or the second predetermined pulse width so that one different reflection is received by the optical sensor (6);
A method comprising: In the eye tracking system (100):
- at least one projection means (3) having a substantially small numerical aperture - said projection means (3) being configured to project a light beam (4) onto a predetermined area of the cornea (2) of the eye (1), preferably operated in a pulsing mode -;
- at least one optical sensor (6) capable of detecting light (5) reflected from said eye (1) and determining a detection position thereof - preferably said optical sensor (6) comprises a plurality of sensing units (7) forming an array of pixels, preferably each sensing unit (7) comprising a photodetector (8), preferably each photodetector being a single photon avalanche detector (SPAD), wherein said light (5) reflected from said eye (1) comprises at least one, preferably two specular reflections, preferably at least a first Purkinje reflection (P1), more preferably at least a first Purkinje reflection (P1) and a fourth Purkinje reflection (P4), and said system (100) is configured to calculate a direction and/or a position and/or a change thereof of said eye (1) based on a detection position and/or a displacement thereof of said at least one reflection;
Including,
wherein said at least one projection means (3) is one movable projection means positioned so that said at least one reflection is received by said optical sensor (6), or
An eye tracking system (100), wherein said at least one projection means (3) is preferably a plurality of projection means (3) arranged in a one-dimensional or two-dimensional array, and wherein said system (100) is configured to find at least one projection means that causes said at least one reflection, preferably at least a first Purkinje reflection (P1) and a fourth Purkinje reflection (P4), to be received by said optical sensor (6). An eye tracking system (100) according to claim 12, wherein during calibration of the system (100), the system is configured to determine a ratio between a displacement of a detection position observed by the optical sensor (6) and an angular movement of the eye (1). An eye tracking system (100) according to claim 12 or 13, wherein said at least one optical sensor (6) is configured to simultaneously measure at least two, preferably two, detection positions on said optical sensor (6) and provide said detection positions to an output of said sensor (6), each of said detection positions corresponding to a different specular reflection.