CN118918139A - Eyeball tracking processing method, device, equipment and storage medium - Google Patents

Abstract

The present invention discloses a processing method, device, equipment and storage medium for eye tracking. It includes: determining the current eye tracking related scene; determining the camera output mode and light source wavelength according to the eye tracking related scene; wherein the camera output mode includes at least one of the following: full pixel output mode, region of interest ROI output mode, downsampling output mode, ROI superposition downsampling output mode; the light source wavelength includes a first wavelength and a second wavelength; based on the camera output mode and the light source wavelength, the data in the eye tracking related scene is processed. The eye tracking processing method provided by the embodiment of the present invention selects the corresponding camera output mode and/or light source wavelength in different eye tracking scenes, which can not only improve the accuracy of eye tracking, but also increase the adaptability of eye tracking.

Description

Eye tracking processing method, device, equipment and storage medium

Technical Field

Embodiments of the present invention relate to the field of eye tracking technology, and in particular, to a processing method, device, equipment and storage medium for eye tracking.

Background Art

In existing eye tracking technologies, when collecting user eye images, the wavelength of the infrared light source and the image output mode of the camera are both relatively simple. This method cannot be applied to various usage scenarios in the eye tracking process, which not only affects the accuracy of eye tracking, but also makes the applicability of eye tracking poor.

Summary of the invention

The embodiments of the present invention provide an eye tracking processing method, apparatus, device and storage medium, which select corresponding camera output modes and/or light source wavelengths in different eye tracking scenarios, which can not only improve the accuracy of eye tracking, but also increase the adaptability of eye tracking.

In a first aspect, an embodiment of the present invention provides an eye tracking processing method, comprising:

Determine a current eye tracking related scenario; wherein the eye tracking related scenario includes at least one of the following: a wearable eye tracking device wearing scenario, a biometric registration scenario, an eye tracking calibration scenario, a first eye tracking scenario, a second eye tracking scenario, and a third eye tracking scenario;

Determine the camera output mode and the light source wavelength according to the eye tracking related scene; wherein the camera output mode includes at least one of the following: full pixel output mode, region of interest ROI output mode, downsampling output mode, ROI superposition downsampling output mode; the light source wavelength includes a first wavelength and a second wavelength;

The data in the eye tracking related scene is processed based on the camera output mode and the light source wavelength.

In a second aspect, an embodiment of the present invention further provides an eye tracking processing device, comprising:

An eye tracking related scene determination module, used to determine a current eye tracking related scene; wherein the eye tracking related scene includes at least one of the following: a wearable eye tracking device wearing scene, a biometric registration scene, an eye tracking calibration scene, a first eye tracking scene, a second eye tracking scene, and a third eye tracking scene;

A module for determining a camera image output mode and a light source wavelength, for determining a camera image output mode and a light source wavelength according to the eye tracking related scene; wherein the camera image output mode includes at least one of the following: a full pixel image output mode, a region of interest ROI image output mode, a downsampling image output mode, and a ROI superimposed downsampling image output mode; and the light source wavelength includes a first wavelength and a second wavelength;

A data processing module is used to process the data in the eye tracking related scene based on the camera output mode and the light source wavelength.

In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

at least one processor; and

a memory communicatively connected to the at least one processor; wherein,

The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the eye tracking processing method according to any one of claims 1 to 8.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the eye tracking processing method described in the embodiment of the present invention when executed.

The embodiment of the present invention discloses a processing method, device, equipment and storage medium for eye tracking. Determine the current eye tracking related scene; wherein the eye tracking related scene includes at least one of the following: a wearable eye tracking device wearing scene, a biometric registration scene, an eye tracking calibration scene, a first eye tracking scene, a second eye tracking scene and a third eye tracking scene; determine the camera output mode and the light source wavelength according to the eye tracking related scene; wherein the camera output mode includes at least one of the following: a full pixel output mode, a region of interest ROI output mode, a downsampling output mode, and a ROI superimposed downsampling output mode; the light source wavelength includes a first wavelength and a second wavelength; based on the camera output mode and the light source wavelength, process the data in the eye tracking related scene. The eye tracking processing method provided by the embodiment of the present invention selects the corresponding camera output mode and/or light source wavelength in different eye tracking scenes, which can not only improve the accuracy of eye tracking, but also increase the adaptability of eye tracking.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of an eye tracking processing method in Embodiment 1 of the present invention;

FIG2 is a flowchart of data processing in a scenario where a wearable eye tracking device is worn in Embodiment 1 of the present invention;

FIG3 is a data processing flow chart of a biometric registration scenario in the first embodiment of the present invention;

FIG4 is a data processing flow chart of an eye tracking calibration scenario in Embodiment 1 of the present invention;

FIG5 is a data processing flow chart of a first eye tracking scenario in Embodiment 1 of the present invention;

FIG6 is a data processing flow chart of a second eye tracking scenario in Embodiment 1 of the present invention;

7 is a data processing flow chart of a third eye tracking scenario in the first embodiment of the present invention;

FIG8 is a schematic diagram of the structure of an eye tracking processing device in Embodiment 2 of the present invention;

FIG. 9 is a schematic diagram of the structure of an electronic device in Embodiment 3 of the present invention.

DETAILED DESCRIPTION

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

Embodiment 1

FIG1 is a flow chart of an eye tracking processing method provided by Embodiment 1 of the present invention. This embodiment is applicable to the case of processing data in eye tracking. The method can be executed by an eye tracking processing device, which can be implemented in the form of software and/or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc. Specifically, it includes the following steps:

S110, determining a current eye tracking related scene.

Among them, eye tracking related scenarios include at least one of the following: a wearable eye tracking device wearing scenario, a biometric registration scenario, an eye tracking calibration scenario, a first eye tracking scenario, a second eye tracking scenario and a third eye tracking scenario.

In this embodiment, the eye tracking related scenes can be selected according to the actual needs of the user for the eye tracking device. If the user is using the eye tracking device for the first time, the user needs to enter the wearable eye tracking device wearing scene, the biometric registration scene and the eye tracking calibration scene in sequence. After the calibration is completed, the user can select the first eye tracking scene, the second eye tracking scene or the third eye tracking scene for eye tracking. In different eye tracking related scenes, the corresponding camera output mode and light source wavelength can be selected to meet the data processing requirements of the current eye tracking related scene.

Among them, the wearable eye tracking device wearing scenario can be understood as a scenario in which the eye tracking device is adjusted during the wearing process. The biometric registration scenario can be understood as the process of storing the user's biometrics in a database. The eye tracking calibration scenario can be understood as the process of obtaining the user calibration coefficient. The first eye tracking scenario can be understood as an eye tracking scenario of ordinary accuracy. The second eye tracking scenario can be understood as an eye tracking scenario of higher accuracy. The third eye tracking scenario can be understood as a low-power and low-precision eye tracking scenario.

S120, determining a camera output mode and a light source wavelength according to the eye tracking related scene.

Among them, the camera output mode includes at least one of the following: full pixel output mode, region of interest ROI output mode, downsampling output mode, ROI superposition downsampling output mode; the light source wavelength includes a first wavelength and a second wavelength.

Among them, the full pixel output mode can be understood as outputting all pixels collected by the camera, and the pixel of the image output by the camera is the highest. For example, the output image can be 15 million pixels. The region of interest (ROI) output mode can be to outline the area to be processed in the image collected by the camera in the form of a box, circle, ellipse, irregular polygon, etc., which is called the region of interest, and output the image corresponding to the region of interest. The pixel of the image output by the ROI output mode is smaller than the pixel of the full pixel output mode. For example, the output image can be 1.6 million pixels. The downsampling output mode can be a Binning mode or a Skipping mode. The Binning mode can be understood as adding the charges induced by adjacent pixels together and reading them out in a pixel mode. Binning is divided into horizontal Binning and vertical Binning. Horizontal Binning is to add the charges of adjacent rows together and read them out, while vertical Binning is to add the charges of adjacent columns together and read them out. The Skipping mode can be understood as extracting the specified pixel points for output. The pixels of the image output by the downsampling output mode are smaller than those of the full pixel output mode. For example, the output image can be 1.6 million pixels. The ROI overlay downsampling output mode can be understood as using the two output modes at the same time to produce an overlay effect. The image output by this output mode has smaller pixels and smaller data volume. It is generally used for low-power and low-precision eye tracking.

In this embodiment, the first wavelength and the second wavelength are in the range of 800nm-1000nm, and the first wavelength and the second wavelength may be the same or different. Both can meet the technical requirements of eye tracking and biometric recognition. However, preferably, the first wavelength is smaller than the second wavelength. For example, the first wavelength may be 810nm or 850nm, and the second wavelength may be 940nm. Of course, the first wavelength and the second wavelength may also be other values.

Specifically, the process of determining the camera image output mode and the wavelength of the light source according to the eye tracking related scene can be: if the eye tracking related scene is a wearable eye tracking device wearing scene, the determined camera image output mode is the downsampling image output mode, and the determined light source wavelength is the second wavelength (i.e., 940nm). If the eye tracking related scene is a biometric registration scene, the determined camera image output mode is the full pixel image output mode or the ROI image output mode, and the determined light source wavelength is the first wavelength (i.e., 810nm or 850nm). If the eye tracking related scene is an eye tracking calibration scene, the determined camera image output mode is the ROI image output mode or the full pixel image output mode, and the determined light source wavelength is the first wavelength. If the eye tracking related scene is the first eye tracking scene, the determined camera image output mode is the ROI image output mode and the downsampling image output mode, and the determined light source wavelength is the first wavelength and the second wavelength. If the eye tracking related scene is the second eye tracking scene, the determined camera image output mode is the ROI image output mode, and the determined light source wavelength is the first wavelength and the second wavelength. If the eye tracking related scene is the third eye tracking scene, the determined camera output mode is the ROI output mode and the ROI superposition downsampling output mode, and the determined light source wavelength is the first wavelength or the first wavelength and the second wavelength.

S130, processing data in eye tracking related scenarios based on the camera output mode and the light source wavelength.

In this embodiment, in different eye-tracking related scenarios, the selected camera output modes and light source wavelengths are different. Based on the determined camera output modes and light source wavelengths, the data in the eye-tracking related scenarios are processed to realize the functions in the eye-tracking related scenarios.

Optionally, if the eye tracking related scenario is a scenario in which a wearable eye tracking device is worn, the method for processing data in the eye tracking related scenario based on the camera output mode and the light source wavelength may be: capturing a first eye image of the user based on the downsampling output mode and a light source of a second wavelength; and adjusting the position and/or pupil distance of the wearable eye tracking device based on the first eye image.

Among them, collecting the first eye image of the user based on the downsampling output mode and the light source of the second wavelength can be understood as: controlling the light source of the second wavelength to illuminate the user's eyes, and then controlling the camera to collect the user's eye image, and outputting the eye image according to the downsampling output mode, so as to obtain the first eye image. Adjusting the position of the wearable eye tracking device may include adjusting the up and down position and left and right position of the wearable eye tracking device or adjusting it in other directions. Adjusting the pupil distance of the wearable eye tracking device may include adjusting the horizontal spacing of the eyepieces of the wearable eye tracking device. The horizontal spacing adjustment of the eyepieces can be understood as taking the center line of the two eyepieces as a reference, and adjusting the distance symmetrically toward the center line or both sides at the same time. It includes automatic adjustment and manual knob adjustment.

In this embodiment, during the wearing process of the wearable eye tracking device, if the wearable eye tracking device needs to be adjusted, the resolution of the eye image required here can be relatively low. Therefore, selecting a light source with a second wavelength of 940nm and a downsampling output mode can meet the wearing requirements of the wearable eye tracking device and save energy.

Specifically, the process of adjusting the position and/or pupil distance of the wearable eye tracking device based on the eye image can be: determining first eye feature data based on the first eye image; determining first adjustment information and second adjustment information based on the first eye feature data; guiding the user to adjust the position of the wearable eye tracking device based on the first adjustment information, and adjusting the pupil distance of the wearable eye tracking device based on the second adjustment information.

Among them, the first eye feature data includes at least one of the following: pupil position, pupil shape, eyelid position, eye corner position, and may also include line of sight direction. The line of sight direction here is an approximate line of sight direction, which is roughly determined based on the position of the pupil and the pupil shape at the upper and lower eyelids and the inner and outer corners of the eyes. Because in the process of wearing a wearable eye tracking device, it is most accurate to adjust the position and pupil distance of the device when the user is looking directly in front of the screen. Therefore, after roughly determining the user's line of sight direction, if the user is not looking directly in front of the screen, a first prompt message will be issued to prompt the user to look directly in front of the screen (it can be a voice prompt or a text prompt). This step is to prepare for the subsequent device position adjustment.

In order to better perform eye tracking and user experience, the wearable eye tracking device can set an optimal wearing position range. When the user's eyes fall into the optimal wearing position range, it means that the optimal wearing position requirements have been met. This optimal wearing position range may not be displayed on the screen. The system determines whether the user's eye position meets the optimal wearing position requirements based on the first eye feature data, and prompts the user to adjust the position (i.e., the first adjustment information) in the form of voice or text; in another embodiment, the optimal wearing position range can also be displayed on the device screen (for example, two circles representing the left eye and the right eye, or a rectangular frame range, etc.), and the user's eye image will also be displayed in real time on the device screen. The user adjusts the position of the device according to the image information on the screen. In this embodiment, the first adjustment information is the image information displayed on the screen.

Specifically, after obtaining the first eye feature data, it is determined whether the user's eyes fall within the optimal wearing position range according to the first eye feature data, whether the pupil distance of the wearable eye tracking device matches the user's pupil distance, if not, the first adjustment information and the second adjustment information are generated, and the user is guided to adjust the position of the wearable eye tracking device according to the first adjustment information, and the pupil distance of the wearable eye tracking device is adjusted according to the second adjustment information, so that the user's eyes fall within the optimal wearing position range, and the pupil distance of the wearable eye tracking device matches the user's pupil distance, that is, it meets the wearing requirements. Exemplarily, FIG2 is a data processing process in the wearable eye tracking device wearing scenario in this embodiment. As shown in FIG2, after the user wears the wearable eye tracking device and turns on the wearable eye tracking device; turns on the 940nm light source to illuminate the eyes, and starts the eye tracking camera to shoot the user's eyes, and outputs the eye image in a downsampling output mode; determines the eye feature data according to the eye image, and guides the user to adjust the position of the wearable eye tracking device and/or adjust the pupil distance of the wearable eye tracking device based on the eye feature data.

It should be noted that the position adjustment and pupil distance adjustment of the wearable eye tracking device can be a unified adjustment process, that is, the position adjustment of the wearable eye tracking device is performed first and then the pupil distance of the wearable eye tracking device is adjusted; or it can be two independent adjustment processes.

Optionally, if the eye tracking related scene is a biometric registration scene, the process of processing the data in the eye tracking related scene based on the camera output mode and the light source wavelength can be: based on the full pixel output mode or the ROI output mode and the light source of the first wavelength, collecting eye images of the user when looking in different directions, to obtain multiple second eye images; extracting local biometric features from the multiple second eye images respectively; splicing multiple local biometric features to obtain target biometric features; and storing the target biometric features.

Among them, the biometric features include at least one of the following: iris features, fundus features, retinal vascular features, vascular features on the eye pattern sclera, and eye shape. In the biometric registration scenario, high-precision biometric features must be extracted, so the accuracy of the eye image is required to be high, and the ambient brightness is also required to be higher. Therefore, the wavelength of the light source is selected to be 810/850nm. The brightness of the infrared light in this band will be higher than that of the infrared light of 940nm. The camera output mode selects the full pixel output mode or the ROI output mode. Specifically, guide points are displayed in different directions on the display interface (UI) of the wearable eye tracking device to guide the user to turn his eyes to look in different directions, and the eye images when the user looks at the guide points are collected to obtain multiple second eye images, and then the local biometric features of the user are extracted from the multiple second eye images respectively, and finally the multiple local biometric features are spliced to obtain the target biometric features, and the target biometric features are stored to complete the registration of biometric features.

Exemplarily, FIG3 is a data processing process in a biometric registration scenario. As shown in FIG3 , the process is: the user completes the wearing adjustment of the wearable eye tracking device and meets the wearing requirements; turns on the 810/850nm light source to illuminate the eyes, starts the eye tracking camera to shoot the user's eyes, and outputs the eye image in full-pixel output mode; guides the user to rotate the eyeballs to look in different directions through the UI interface, and collects eye images when the user looks in different directions; extracts local biometric features of eye data in different directions, splices multiple local biometric features, and obtains target biometric features; stores the target biometric features to complete the registration of biometric features.

Optionally, if the eye tracking related scene is an eye tracking calibration scene, the method of processing the data in the eye tracking related scene based on the camera output mode and the wavelength of the light source can be: based on the ROI output mode or the full-pixel output mode and the light source of the first wavelength, a third eye image of the user gazing at the set calibration point is captured; and the calibration coefficient is determined based on the third eye image.

Among them, based on the ROI output mode and the first wavelength light source to collect the third eye image of the user looking at the set calibration point can be understood as: controlling the first wavelength light source to irradiate the user's eyes, and then controlling the camera to collect the user's eye image, and output the eye image according to the ROI output mode or the full pixel output mode, so as to obtain the third eye image. In this embodiment, if the ROI output mode is adopted, when the user's eye tracking calibration is performed, it is not necessary to analyze the entire eye image, only the area of interest in the eye image needs to be analyzed. Therefore, the use of the ROI output mode and the 810/850nm light source will not affect the calibration accuracy, but also reduce the amount of calculation.

Specifically, the process of determining the calibration coefficient based on the third eye image may be: determining third eye feature data according to the third eye image; determining the calibration coefficient based on the third eye feature data; accordingly, after determining the calibration coefficient based on the third eye image, it also includes the following step: binding the calibration coefficient and biometric features and storing them.

Among them, the third eye feature data includes at least one of the following: fundus data, pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, and light spot position. In this embodiment, the calibration point is set on the display interface of the wearable eye tracking device, and then the third eye image of the user gazing at the set calibration point is collected based on the ROI output mode or the full pixel output mode and the light source of the first wavelength, and the fundus data, pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, light spot position and other eye feature data are extracted from the third eye image, and finally the calibration coefficient corresponding to each set calibration point is determined based on the eye feature data, and the calibration coefficient is bound to the biometric feature and stored.

Exemplarily, FIG4 is a data processing flow in the eye tracking calibration scenario in this embodiment. As shown in FIG4 , the flow is: the user completes the wearing adjustment of the wearable eye tracking device, turns on the 810/850nm light source to illuminate the eyes, and starts the eye tracking camera to shoot the user's eyes, and outputs the eye image in ROI output mode; the set calibration point is displayed to collect the eye image of the user looking at the set calibration point, and the fundus data, pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, light spot position and other eye feature data in the eye image are extracted; the calibration coefficient corresponding to each set calibration point is determined based on the eye feature data; the calibration coefficient is bound to the biometric feature and stored to complete the calibration.

Optionally, if the eye tracking related scene is the first eye tracking scene, which is a normal precision eye tracking scene, the process of processing the data in the eye tracking related scene based on the camera output mode and the light source wavelength may be: based on the ROI output mode and the light source of the first wavelength, collecting the fourth eye image of the user; determining the user's biometric characteristics based on the fourth eye image; obtaining the pre-stored user's calibration coefficient based on the biometric characteristics; switching the camera output mode to the downsampling output mode, and switching the light source wavelength to the second wavelength; based on the downsampling output mode and the light source of the second wavelength, collecting the user's fifth eye image; and performing eye tracking on the user based on the fifth eye image and the calibration coefficient.

In this embodiment, when performing eye tracking, the user's biometrics must first be obtained to obtain the user's pre-stored calibration coefficients from the database based on the biometrics. Here, the accuracy of the eye image is required to be high, so the ROI output mode and the first wavelength light source are selected to collect the eye image. After obtaining the calibration coefficient, when performing eye tracking based on the calibration coefficient, if the accuracy of eye tracking is not required, the camera output mode can be switched to the downsampling output mode, and the light source wavelength can be switched to the second wavelength to perform eye tracking.

Among them, the method of tracking the user's eyes based on the fifth eye image and the calibration coefficient can refer to the existing gaze point or gaze direction estimation algorithm, which is not limited here.

Exemplarily, FIG5 is a data processing process in the first eye tracking scenario in this embodiment, which includes: the user completes the wearing adjustment of the wearable eye tracking device; turns on the 810/850nm light source to illuminate the eyes, starts the eye tracking camera to shoot the user's eyes, and outputs the eye image in ROI output mode; determines the user's biometrics based on the eye image, and obtains the pre-stored user's calibration coefficient based on the biometrics; switches the camera output mode to the downsampling output mode, switches the light source wavelength to 940nm, and collects the user's eye image again; performs eye tracking based on the eye image and the calibration coefficient.

It should be noted that if the first wavelength and the second wavelength are equal, there is no need to switch the light source.

Optionally, if the eye tracking related scene is the second eye tracking scene, which is a high-precision eye tracking scene, the process of processing the data in the eye tracking related scene based on the camera output mode and the light source wavelength may be: based on the ROI output mode and the light source of the first wavelength, a sixth eye image of the user is collected; based on the sixth eye image, the user's biometric characteristics are determined; based on the biometric characteristics, a pre-stored calibration coefficient of the user is obtained; the light source wavelength is switched to the second wavelength, and based on the ROI output mode and the light source of the second wavelength, a seventh eye image of the user is collected; and eye tracking of the user is performed based on the seventh eye image and the calibration coefficient.

In this embodiment, when performing eye tracking, the user's biometrics must first be obtained to obtain the user's pre-stored calibration coefficients from the database based on the biometrics. Here, the accuracy of the eye image is required to be high, so the ROI output mode and the first wavelength light source are selected to collect the eye image. After obtaining the calibration coefficient, when performing eye tracking based on the calibration coefficient, if the accuracy of eye tracking is required to be high, the ROI output mode is continued to be used and the second wavelength light source is used to collect the eye image, and eye tracking is performed based on the re-collected eye image and the calibration coefficient.

Exemplarily, FIG6 is the data processing process in the second eye tracking scenario in this embodiment, which includes: the user completes the wearing adjustment of the wearable eye tracking device; turns on the 810/850nm light source to illuminate the eyes, and starts the eye tracking camera to shoot the user's eyes, and outputs the eye image in ROI output mode; determines the user's biological characteristics based on the eye image, and obtains the pre-stored user's calibration coefficient based on the biological characteristics; switches the light source wavelength to 940nm, and collects the user's eye image again; performs eye tracking based on the eye image and the calibration coefficient. It should be noted that if the first wavelength and the second wavelength are equal, there is no need to switch the light source.

Optionally, if the eye tracking related scenario is a third eye tracking scenario, which is a low-power and low-precision eye tracking scenario, the process of processing the data in the eye tracking related scenario based on the camera output mode and the light source wavelength may be: based on the ROI output mode and the light source of the first wavelength, collecting the eighth eye image of the user; determining the user's biometric characteristics based on the eighth eye image; obtaining the pre-stored user's calibration coefficient based on the biometric characteristics; switching the camera output mode to the ROI overlay downsampling output mode, switching the light source wavelength to the second wavelength or still using the first wavelength; collecting the user's ninth eye image based on the ROI overlay downsampling output mode and the light source of the first wavelength or the second wavelength; and performing eye tracking on the user based on the ninth eye image and the calibration coefficient.

In this embodiment, when performing eye tracking, the user's biometrics must first be obtained to obtain the pre-stored user calibration coefficients from the database based on the biometrics. Here, the accuracy requirements for the eye image are relatively high, so the ROI output mode and the first wavelength light source are selected to collect the eye image. After obtaining the calibration coefficient, when performing eye tracking based on the calibration coefficient, because this mode is a low-power and low-precision eye tracking mode, the camera output mode can be switched to the ROI superimposed downsampling output mode, and the wavelength of the light source remains unchanged or switched to the second wavelength for eye tracking. Since the output eye image has lower pixels and less data is transmitted, power consumption can be saved.

Among them, the method of tracking the user's eyes based on the ninth eye image and the calibration coefficient can refer to the existing gaze point or gaze direction estimation algorithm, which is not limited here.

Exemplarily, FIG7 is a data processing process in the third eye tracking scenario in this embodiment, which includes: the user completes the wearing adjustment of the wearable eye tracking device; turns on the 810/850nm light source to illuminate the eyes, starts the eye tracking camera to shoot the user's eyes, and outputs the eye image in ROI output mode; determines the user's biometrics based on the eye image, and obtains the pre-stored user's calibration coefficient based on the biometrics; switches the camera output mode to the ROI superposition downsampling output mode, keeps the light source wavelength unchanged or switches the light source wavelength to 940nm, and collects the user's eye image; performs eye tracking based on the eye image and the calibration coefficient.

It should be noted that if the first wavelength and the second wavelength are equal, there is no need to switch the light source.

The technical solution of this embodiment determines the current eye tracking related scene; wherein the eye tracking related scene includes at least one of the following: a wearable eye tracking device wearing scene, a biometric registration scene, an eye tracking calibration scene, a first eye tracking scene, a second eye tracking scene, and a third eye tracking scene; the camera output mode and the light source wavelength are determined according to the eye tracking related scene; wherein the camera output mode includes: a full pixel output mode, a region of interest ROI output mode, a downsampling output mode, and a ROI superposition downsampling output mode; the light source wavelength includes a first wavelength and a second wavelength; and the data in the eye tracking related scene is processed based on the camera output mode and the light source wavelength. The eye tracking processing method provided by the embodiment of the present invention selects the corresponding camera output mode and/or light source wavelength in different eye tracking scenes, which can not only improve the accuracy of eye tracking, but also increase the adaptability of eye tracking.

Embodiment 2

FIG8 is a schematic diagram of the structure of an eye tracking processing device provided in Embodiment 2 of the present invention. As shown in FIG8 , the device includes:

The eye tracking related scene determination module 810 is used to determine the current eye tracking related scene; wherein the eye tracking related scene includes at least one of the following: a wearable eye tracking device wearing scene, a biometric registration scene, an eye tracking calibration scene, a first eye tracking scene, a second eye tracking scene, and a third eye tracking scene;

The image output mode and light source wavelength determination module 820 is used to determine the camera image output mode and light source wavelength according to the eye tracking related scene; wherein the camera image output mode includes at least one of the following: full pixel image output mode, region of interest ROI image output mode, downsampling image output mode, ROI superposition downsampling image output mode; the light source wavelength includes a first wavelength and a second wavelength;

The data processing module 830 is used to process the data in the eye tracking related scenes based on the camera output mode and the light source wavelength.

Optionally, the image output mode and light source wavelength determination module 820 is further used to:

If the eye tracking related scene is a wearable eye tracking device wearing scene, the determined camera image output mode is a downsampling image output mode, and the determined light source wavelength is the second wavelength;

Optionally, the data processing module 830 is further configured to:

Collecting a first eye image of the user based on a downsampling image output mode and a light source of a second wavelength;

The position and/or pupil distance of the wearable eye tracking device are adjusted based on the first eye image.

Optionally, the data processing module 830 is further configured to:

Determine first eye feature data according to the first eye image; wherein the first eye feature data includes at least one of the following: pupil position, pupil shape, eyelid position, and eye corner position;

Determine first adjustment information and/or second adjustment information according to the first eye feature data;

The user is guided to adjust the position of the wearable eye tracking device according to the first adjustment information, and/or the pupil distance of the wearable eye tracking device is adjusted according to the second adjustment information.

Optionally, the image output mode and light source wavelength determination module 820 is further used to:

If the eye tracking related scene is a biometric registration scene, the determined camera output mode is a full pixel output mode or a ROI output mode, and the determined light source wavelength is a first wavelength;

Optionally, the data processing module 830 is further configured to:

Based on the full pixel output mode or the ROI output mode and the light source of the first wavelength, the eye images of the user when gazing in different directions are collected to obtain a plurality of second eye images;

respectively extracting local biological features from a plurality of second eye images;

Splicing multiple local biometric features to obtain target biometric features;

The target biometric features are stored; wherein the biometric features include at least one of the following: iris features, fundus features, retinal vascular features, vascular features on eye patterns and sclera, and eye shape.

Optionally, the image output mode and light source wavelength determination module 820 is further used to:

If the eye tracking related scene is an eye tracking calibration scene, the determined camera image output mode is the ROI image output mode or the full pixel image output mode, and the determined light source wavelength is the first wavelength;

Optionally, the data processing module 830 is further configured to:

Based on the ROI output mode or the full pixel output mode and the light source of the first wavelength, a third eye image of the user gazing at the set calibration point is collected;

Calibration coefficients are determined based on the third eye image.

Optionally, the data processing module 830 is further configured to:

Determine third eye feature data according to the third eye image; wherein the third eye feature data includes at least one of the following: fundus data, pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, and light spot position;

determining a calibration coefficient based on the third eye feature data;

The calibration coefficients are bound to the biometric features and stored.

Optionally, the image output mode and light source wavelength determination module 820 is further used to:

If the eye tracking related scene is the first eye tracking scene, the determined camera output mode is the ROI output mode, and the determined light source wavelength is the first wavelength;

Optionally, the data processing module 830 is further configured to:

Collecting a fourth eye image of the user based on the ROI output mode and the light source of the first wavelength;

determining a biometric characteristic of the user based on the fourth eye image;

Obtaining a pre-stored user calibration coefficient based on biometrics;

Switch the camera output mode to downsampling output mode, and switch the light source wavelength to the second wavelength;

Collecting a fifth eye image of the user based on the downsampling image output mode and the light source of the second wavelength;

Eye tracking of the user is performed based on the fifth eye image and the calibration coefficients.

Optionally, the image output mode and light source wavelength determination module 820 is further used to:

If the eye tracking related scene is the second eye tracking scene, the determined camera output mode is the ROI output mode, and the determined light source wavelength is the first wavelength;

Optionally, the data processing module 830 is further configured to:

Collecting a sixth eye image of the user based on the ROI output mode and the light source of the first wavelength;

determining a biometric characteristic of the user based on the sixth ocular image;

Obtaining a pre-stored user calibration coefficient based on biometrics;

Collecting a seventh eye image of the user based on the ROI output mode and the light source of the second wavelength;

Eye tracking of the user is performed based on the seventh eye image and the calibration coefficients.

Optionally, the image output mode and light source wavelength determination module 820 is further used to:

If the eye tracking related scene is the third eye tracking scene, the determined camera output mode is the ROI output mode and the ROI superposition downsampling output mode, and the determined light source wavelength is the first wavelength or the first wavelength and the second wavelength;

Optionally, the data processing module 830 is further configured to:

Collecting an eighth eye image of the user based on the ROI output mode and the light source of the first wavelength;

determining a biometric characteristic of the user based on an eighth eye image;

Obtaining a pre-stored user calibration coefficient based on biometrics;

Collecting a ninth eye image of the user based on the ROI superposition downsampling output mode and the light source of the first wavelength or the second wavelength;

Eye tracking of the user is performed based on the ninth eye image and the calibration coefficient.

The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

Embodiment 3

FIG9 shows a block diagram of an electronic device 10 that can be used to implement an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, XR glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and/or required herein.

As shown in FIG9 , the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11 in communication, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to the bus 14.

A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the processing method of eye tracking.

In some embodiments, the eye tracking processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the eye tracking processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the eye tracking processing method in any other appropriate manner (e.g., by means of firmware).

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions/operations specified in the flow chart and/or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (12)

1. A method for processing eye tracking, characterized by comprising: Determine a current eye tracking related scenario; wherein the eye tracking related scenario includes at least one of the following: a wearable eye tracking device wearing scenario, a biometric registration scenario, an eye tracking calibration scenario, a first eye tracking scenario, a second eye tracking scenario, and a third eye tracking scenario; Determine the camera output mode and the light source wavelength according to the eye tracking related scene; wherein the camera output mode includes at least one of the following: full pixel output mode, region of interest ROI output mode, downsampling output mode, ROI superposition downsampling output mode; the light source wavelength includes a first wavelength and a second wavelength; The data in the eye tracking related scene is processed based on the camera output mode and the light source wavelength. 2. The method according to claim 1, characterized in that determining the camera output mode and the light source wavelength according to the eye tracking related scene comprises: If the eye tracking related scene is a wearing scene of the wearable eye tracking device, the determined camera image output mode is the downsampling image output mode, and the determined light source wavelength is the second wavelength; Accordingly, the data in the eye tracking related scene is processed based on the camera output mode and the light source wavelength, including: Collecting a first eye image of the user based on the downsampling image output mode and the light source of the second wavelength; The position and/or pupil distance of the wearable eye tracking device are adjusted based on the first eye image. 3. The method according to claim 2, characterized in that adjusting the position and/or pupil distance of the wearable eye tracking device based on the eye image comprises: Determine first eye feature data according to the first eye image; wherein the first eye feature data includes at least one of the following: pupil position, pupil shape, eyelid position, and eye corner position; Determining first adjustment information and/or second adjustment information according to the first eye feature data; The user is guided to adjust the position of the wearable eye tracking device according to the first adjustment information, and/or the pupil distance of the wearable eye tracking device is adjusted according to the second adjustment information. 4. The method according to claim 1, characterized in that determining the camera output mode and the light source wavelength according to the eye tracking related scene comprises: If the eye tracking related scene is a biometric registration scene, the determined camera image output mode is the full pixel image output mode or the ROI image output mode, and the determined light source wavelength is the first wavelength; Accordingly, the data in the eye tracking related scene is processed based on the camera output mode and the light source wavelength, including: Based on the full-pixel image output mode or the ROI image output mode and the light source of the first wavelength, eye images of the user when gazing in different directions are collected to obtain a plurality of second eye images; respectively extracting local biological features from the plurality of second eye images; splicing the plurality of local biometric features to obtain a target biometric feature; The target biometric features are stored; wherein the biometric features include at least one of the following: iris features, fundus features, retinal vascular features, vascular features on eye wrinkles and sclera, and eye shape. 5. The method according to claim 1, characterized in that determining the camera output mode and the light source wavelength according to the eye tracking related scene comprises: If the eye tracking related scene is an eye tracking calibration scene, the determined camera image output mode is the ROI image output mode or the full pixel image output mode, and the determined light source wavelength is the first wavelength; Accordingly, the data in the eye tracking related scene is processed based on the camera output mode and the light source wavelength, including: Based on the ROI image output mode or the full-pixel image output mode and the light source of the first wavelength, a third eye image of the user gazing at a set calibration point is collected; Calibration coefficients are determined based on the third eye image. 6. The method according to claim 5, characterized in that determining the calibration coefficient based on the third eye image comprises: Determine third eye feature data according to the third eye image; wherein the third eye feature data includes at least one of the following: fundus data, pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, and light spot position; determining a calibration coefficient based on the third eye feature data; Accordingly, after determining the calibration coefficient based on the third eye image, the method further includes: The calibration coefficient is bound to the biometric feature and then stored. 7. The method according to claim 1, characterized in that determining the camera output mode and the light source wavelength according to the eye tracking related scene comprises: If the eye tracking related scene is the first eye tracking scene, the determined camera output mode is the ROI output mode and the downsampling output mode, and the determined light source wavelength is the first wavelength and the second wavelength; Accordingly, the data in the eye tracking related scene is processed based on the camera output mode and the light source wavelength, including: Collecting a fourth eye image of the user based on the ROI output mode and the light source of the first wavelength; determining a biometric feature of the user based on the fourth eye image; Acquire a pre-stored calibration coefficient of the user according to the biometric feature; Switching the camera image output mode to the downsampling image output mode, and switching the light source wavelength to the second wavelength; Collecting a fifth eye image of the user based on the downsampling image output mode and the light source of the second wavelength; Eye tracking of the user is performed based on the fifth eye image and the calibration coefficient. 8. The method according to claim 1, characterized in that determining the camera output mode and the light source wavelength according to the eye tracking related scene comprises: If the eye tracking related scene is the second eye tracking scene, the determined camera output mode is the ROI output mode, and the determined light source wavelength is the first wavelength and the second wavelength; Accordingly, the data in the eye tracking related scene is processed based on the camera output mode and the light source wavelength, including: Collecting a sixth eye image of the user based on the ROI output mode and the light source of the first wavelength; determining a biometric feature of the user based on the sixth eye image; Acquire a pre-stored calibration coefficient of the user according to the biometric feature; Switching the wavelength of the light source to the second wavelength; Collecting a seventh eye image of the user based on the ROI output mode and the light source of the second wavelength; Eye tracking of the user is performed based on the seventh eye image and the calibration coefficient. 9. The method according to claim 1, characterized in that determining the camera output mode and the light source wavelength according to the eye tracking related scene comprises: If the eye tracking related scene is the third eye tracking scene, the determined camera image output mode is the ROI image output mode and the ROI superposition downsampling image output mode, and the determined light source wavelength is the first wavelength or the first wavelength and the second wavelength; Accordingly, the data in the eye tracking related scene is processed based on the camera output mode and the light source wavelength, including: Collecting an eighth eye image of the user based on the ROI output mode and the light source of the first wavelength; determining a biometric feature of the user based on the eighth eye image; Acquire a pre-stored calibration coefficient of the user according to the biometric feature; Switch the camera image output mode to the ROI stacking downsampling image output mode, and switch the light source wavelength to the second wavelength or still use the first wavelength; Collecting a ninth eye image of the user based on the ROI superimposed downsampling image output mode and the light source of the first wavelength or the second wavelength; Eye tracking of the user is performed based on the ninth eye image and the calibration coefficient. 10. An eye tracking processing device, comprising: An eye tracking related scene determination module, used to determine a current eye tracking related scene; wherein the eye tracking related scene includes at least one of the following: a wearable eye tracking device wearing scene, a biometric registration scene, an eye tracking calibration scene, a first eye tracking scene, and a second eye tracking scene; The image output mode and light source wavelength determination module is used to determine the camera image output mode and light source wavelength according to the eye tracking related scene; wherein the camera image output mode includes at least one of the following: full pixel image output mode, region of interest ROI image output mode, downsampling image output mode, ROI superposition downsampling image output mode; the light source wavelength includes a first wavelength and a second wavelength; A data processing module is used to process the data in the eye tracking related scene based on the camera output mode and the light source wavelength. 11. An electronic device, characterized in that the electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the eye tracking processing method according to any one of claims 1 to 8. 12. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the eye tracking processing method according to any one of claims 1 to 8 when executed.