CN116704589B - A gaze point estimation method, electronic device and computer readable storage medium - Google Patents

Abstract

The application provides a gaze point estimation method, electronic equipment and a computer readable storage medium, and relates to the technical field of computers. In the scheme, even if the electronic equipment shoots in an excessively dark or excessively bright environment of a scene, two-dimensional and three-dimensional information of an eye image and eye key points of a user can be acquired, and the gazing information of the user on a display screen can be accurately determined. The method comprises the following steps: acquiring an infrared image and a depth image of a user through a preset camera; identifying an infrared image to obtain a first eye region image of a user and first position information of eye key points of the user in the first eye region image; obtaining a first gaze feature for indicating eye and gaze correlation of the user based on the first eye region image; obtaining second gazing features for indicating three-dimensional features of eye key points of the user from the depth image based on the first position information; and determining the fixation information of the user on the display screen according to the first fixation characteristic and the second fixation characteristic.

Description

A gaze point estimation method, electronic device and computer readable storage medium

Technical Field

The present application relates to the field of computer technology, and in particular to a gaze point estimation method, an electronic device, and a computer-readable storage medium.

Background technique

The fixation point refers to a certain point of the target object that the user's line of sight is aimed at during the visual perception process. At present, electronic devices can assist users in their work. In the process of electronic devices assisting users in their work, the electronic devices can determine the user's fixation point based on the user's activities, and thus assist the user in their work based on the user's fixation point. For example, in a scenario where a user is reading an e-book, the mobile phone can determine the user's fixation point, and determine whether the user has the intention to turn the page based on the user's fixation point. If so, the mobile phone actively provides the user with a page-turning operation. For another example, in a scenario where a user is browsing a web page, the mobile phone can determine the user's fixation point, and determine the content that the user is interested in based on the user's fixation point, and actively recommend the content that the user is interested in to the user.

When an electronic device assists a user in their work, the user needs to face the electronic device so that the electronic device can obtain the user's eye geometry features (or key point information of the eye, such as eye contour, eye canthus opening, pupil direction, etc.), and predict the user's gaze point based on the obtained eye geometry features. However, in actual applications, in some special shooting environments, such as environments where the scene is too dark or too bright, the electronic device cannot obtain the user's eye geometry features, which results in the electronic device being unable to accurately determine the user's gaze point, which in turn causes the electronic device to fail to assist the user in their work, reducing the reliability of the electronic device in assisting the user in their work.

Summary of the invention

The embodiments of the present application provide a gaze point estimation method, an electronic device, and a computer-readable storage medium, which are used to solve the problem that the electronic device in the prior art cannot accurately determine the user's gaze point. Through this solution, the user's gaze point can be determined more accurately.

To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

In a first aspect, a gaze point estimation method is provided, which is applied to an electronic device, wherein the electronic device includes a preset camera and a display screen, and the method includes: collecting an infrared image and a depth image of a user through the preset camera; identifying the infrared image to obtain a first eye area image of the user and first position information of the user's eye key points in the first eye area image; obtaining a first gaze feature based on the first eye area image, wherein the first gaze feature is used to indicate a two-dimensional feature of the user's eye; obtaining a second gaze feature from the depth image based on the first position information, wherein the second gaze feature is used to indicate a three-dimensional feature of the user's eye key points; and determining the user's gaze information on the display screen according to the first gaze feature and the second gaze feature.

In this solution, even in a dark or overly bright scene, the infrared radiation (IR) image captured by the electronic device can also contain a clearer user facial image, and the user's eye area image (i.e., the first eye area image hereinafter) and the position information of the user's eye key points in the user's eye area image can be extracted from the IR image. Since the contents of the user's IR image and the depth image are basically the same, and the positions of the same contents in their respective images are basically the same, the three-dimensional features (i.e., the second gaze feature hereinafter) corresponding to the image for indicating the user's eye key points can be obtained from the depth image through the aforementioned position information of the user's eye key points in the user's eye area image (i.e., the first position information hereinafter). Moreover, the electronic device can also extract the two-dimensional feature information (i.e., the first gaze feature hereinafter) and the second gaze feature for indicating the user's eyes from the eye area image corresponding to the IR image. In this way, the electronic device can determine the user's gaze information on the display screen based on the aforementioned two kinds of gaze feature information. In this way, the electronic device can accurately determine the user's gaze information on the display screen based on the aforementioned two kinds of gaze feature information. In the scenario where the electronic device assists the user in working according to the user's gaze point, the electronic device can successfully assist the user in working, thereby improving the reliability of the electronic device in assisting the user in working.

In a possible implementation manner of the first aspect, the above-mentioned determining the gaze information of the user on the display screen according to the first gaze feature and the second gaze feature includes: obtaining a first eye posture parameter according to the first gaze feature; obtaining a second eye posture parameter according to the second gaze feature; wherein the first eye posture parameter and the second eye posture parameter both include line of sight direction information of the user's eyes, and the line of sight direction information of the user's eyes includes a rotation angle, a pitch angle and a heading angle of the user's eyes in a preset three-dimensional coordinate system; and determining the user's gaze information on the display screen based on the first eye posture parameter and the second eye posture parameter.

Since the first gaze feature is a two-dimensional feature for indicating the key points of the user's eyes, the second gaze feature is obtained based on the second eye region image, and the second gaze feature is a three-dimensional feature for indicating the key points of the user's eyes. The two-dimensional features of the key points of the user's eyes and the three-dimensional features of the key points of the user's eyes can predict the direction of the user's eyes. Therefore, the electronic device can first determine the direction of the user's eyes according to the first gaze feature and the second gaze feature, and then determine the user's gaze information on the display screen based on the direction of the user's eyes.

In a possible implementation manner of the first aspect, the preset three-dimensional coordinate system is a three-dimensional coordinate system with an optical center of a preset camera as its origin.

In a possible implementation of the first aspect, there is a preset correspondence between the first human eye posture parameter, the second human eye posture parameter and the user's gaze information on the display screen, and determining the user's gaze information on the display screen based on the first human eye posture parameter and the second human eye posture parameter includes: using the first human eye posture parameter and the second human eye posture parameter to determine the user's gaze information on the display screen from the preset correspondence.

In a possible implementation manner of the first aspect, before recognizing the infrared image and obtaining the first gaze feature, the method further includes: determining that the infrared image includes a human eye image.

In some cases, the user's head may not be still for a certain period of time, and may turn, look up, or lower the head. In this way, the user's eye image may not exist in the IR image and depth image of the user taken by the electronic device. If the user's eye image does not exist in the IR image and depth image of the user taken by the electronic device, the electronic device will consume a certain amount of energy and time to perform this step. Therefore, in order to save the energy and time consumed by the electronic device to perform this step, the electronic device may also first determine whether the user's eye image exists in the user's IR image. If so, it can then identify the infrared image to obtain the first gaze feature.

In a possible implementation of the first aspect, collecting infrared images and depth images of a user through a preset camera includes: receiving a first operation, the first operation is used to trigger the electronic device to start a preset application; in response to the first operation, after starting the preset application, periodically collecting infrared images and depth images of the user through the preset camera. That is, some applications can be set in the electronic device, and the application can support the gaze point estimation method of the present application, then the electronic device can periodically collect infrared images and depth images of the user through the preset camera after starting the application.

In a possible implementation of the first aspect, the infrared image and depth image of the user are collected through a preset camera, including: if the electronic device is in a preset collaborative working mode, the infrared image and depth image of the user are periodically collected through the preset camera. In this scheme, the collaborative working mode may refer to a state in which the electronic device can determine the user's gaze point, determine the user's intention based on the user's gaze point, and perform corresponding operations based on the user's intention. For example, in a scenario where a user is reading an e-book, the mobile phone can determine the user's gaze point, and determine whether the user has the intention to turn the page based on the user's gaze point. If so, the mobile phone actively provides the user with a page turning operation. If the electronic device is in a collaborative working mode, the electronic device may also periodically collect the user's infrared image and depth image through a preset camera, so that the electronic device can more accurately detect the user's gaze information on the display screen.

In a possible implementation of the first aspect, before collecting the infrared image and depth image of the user through a preset camera, the method also includes: in response to the user's turning on operation of a preset collaborative switch, entering a preset collaborative working mode; wherein the preset collaborative switch is configured in a setting interface of the electronic device, or the preset collaborative switch is configured in a control center of the electronic device, or the preset collaborative switch is configured in a preset application of the electronic device.

Some applications are set in the electronic device, and these applications require the electronic device to start the collaborative working mode and the electronic device periodically collects the user's infrared image and depth image through the preset camera when the conditions for starting the application coexist. Among them, the preset collaborative switch can be configured in the setting interface of the electronic device, or the preset collaborative switch is configured in the control center of the electronic device, or the preset collaborative switch is configured in the preset application of the electronic device. The electronic device can enter the preset collaborative working mode in response to the user's turning on operation of any of the preset collaborative switches. It should be noted that the electronic device can turn on the collaborative working mode before or after starting the application.

In a possible implementation manner of the first aspect, the preset camera is a TOF camera or a 3D structured light camera.

In a possible implementation of the first aspect, the above-mentioned collecting of infrared images and depth images of the user by a preset camera includes: collecting infrared images and depth images by a preset camera when detecting that the user is looking at the display screen. That is, as long as the electronic device detects that the user is looking at the display screen, it can execute the gaze point estimation method of the present solution, thereby increasing the duration of the electronic device executing the gaze point estimation method of the present solution. In this way, in a scenario where the electronic device assists the user in working according to the user's gaze point, the duration of the electronic device accurately assisting the user in working can be increased, thereby improving the user experience.

In a second aspect, a gaze point estimation method is provided, which is applied to an electronic device, wherein the electronic device includes a preset camera and a display screen, and the method includes: collecting an infrared image and a depth image of a user through a preset camera; using the infrared image and the depth image as input, running a human eye gaze information estimation model to obtain the user's gaze information on the display screen; wherein the human eye gaze information estimation model is used to: obtain a first gaze feature of the user's eye area based on identifying the infrared image, and obtain a second gaze feature of the user's eye area based on identifying the depth image, and output the user's gaze information on the display screen; wherein the first gaze feature is used to indicate a two-dimensional feature of the user's eye key point, and the second gaze feature is used to indicate a three-dimensional feature of the user's eye key point.

It can be understood that the model can have a powerful and fast prediction function. By using the human eye gaze information estimation model to obtain the user's gaze information on the display screen, the estimation speed of the gaze information can be improved, and the user experience can be improved. In the scenario where the electronic device assists the user in working according to the user's gaze point, the efficiency of the electronic device in assisting the user in working can be improved, and the user experience can be improved.

In a possible implementation of the second aspect, the above-mentioned human eye gaze information estimation model is also used to: output the user's gaze information on the display screen based on a first gaze feature of the user's eye key points obtained by identifying an infrared image, and a second gaze feature of the user's eye key points obtained by identifying a depth image; wherein the first gaze feature is used to indicate two-dimensional features of the user's eyes related to gaze, and the second gaze feature is used to indicate three-dimensional features of the user's eye key points.

In a possible implementation of the second aspect, there is a preset correspondence between the first eye posture parameter and the second eye posture parameter and the gaze information of the user on the display screen. The running eye gaze information estimation model is also used to: use the first eye posture parameter and the second eye posture parameter to determine the gaze information of the user on the display screen according to the preset correspondence.

In a possible implementation manner of the second aspect, the preset three-dimensional coordinate system is a three-dimensional coordinate system with an optical center of a preset camera as its origin.

In a third aspect, an electronic device is provided, which includes a memory and one or more processors; the memory is used to store code instructions; the processor is used to run the code instructions so that the electronic device executes a gaze point estimation method in any possible design mode of the first aspect and the second aspect.

In a fourth aspect, a computer-readable storage medium is provided, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes a gaze point estimation method in any possible design mode of the first aspect and the second aspect.

In a fifth aspect, a computer program product is provided, comprising a computer program/instruction, which, when executed by a processor, implements the gaze point estimation method in any possible design manner in the first aspect and the second aspect.

Among them, the technical effects brought about by any design method in the second, third, fourth and fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 shows a schematic diagram of an IR image and a depth image;

FIG2 shows a schematic structural diagram of a mobile phone 100;

FIG3 is a schematic diagram showing an application scenario of obtaining a user's face image;

FIG4 shows a schematic flow chart of a method for estimating a gaze point;

FIG5 is a schematic diagram showing a flow chart of a method for estimating a gaze point;

FIG6 shows a calculation schematic diagram of a human eye gaze information estimation model;

FIG7 shows a schematic diagram of an interface of a mobile phone 100;

FIG. 8 shows a schematic diagram of an interface of an e-book application.

Detailed ways

The illustrative embodiments of the present application include, but are not limited to, a gaze point estimation method, an electronic device, and a computer-readable storage medium.

The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

In order to solve the technical problems in the background technology, the embodiment of the present application shows a method for estimating the gaze point, which is applied to an electronic device with a preset camera and a display screen. The method includes: the electronic device collects an infrared (IR) image and a depth image of the user through a preset camera. For example, FIG1 shows a schematic diagram of an IR image and a depth image, such as the IR image and the depth image shown in FIG1. Since the IR image is a modulated infrared light pulse emitted by an infrared light transmitter, which continuously hits the surface of the object and is received by the receiver after reflection, the time difference is calculated by the change of the phase, and then the depth information of the object is calculated in combination with the speed of light. Generally, the photographing effect is not affected by the ambient light. Therefore, even in a dark or overly bright scene, the IR image captured by the electronic device can also have a clearer user facial image, and the user's eye area image (i.e., the first eye area image hereinafter) and the position information of the user's eye key points in the user's eye area image are extracted from the IR image. Since the contents of the IR image and the depth image of the user are basically the same, and the positions of the same contents in the respective images are basically the same, the three-dimensional features (i.e., the second gaze features hereinafter) corresponding to the image and used to indicate the key points of the user's eyes can be obtained from the depth image through the position information of the key points of the user's eyes in the eye area image of the user obtained above (i.e., the first position information hereinafter). Moreover, the electronic device can also extract the two-dimensional feature information (i.e., the first gaze feature hereinafter) and the second gaze feature indicating the user's eyes from the eye area image corresponding to the IR image. In this way, the electronic device can determine the user's gaze information on the display screen based on the aforementioned two types of gaze feature information. The specific scheme for determining the user's gaze information on the display screen based on the two types of gaze feature information will be introduced below.

In this solution, even if the electronic device shoots the user in an environment where the scene is too dark or too bright, the electronic device can obtain the key point information of the two-dimensional features and three-dimensional features of the key points of the user's eyes, and accurately determine the user's gaze information on the display screen. In the scenario where the electronic device assists the user in working according to the user's gaze point, the electronic device can successfully assist the user in working, thereby improving the reliability of the electronic device in assisting the user in working.

Exemplarily, the electronic devices in the embodiments of the present application may be mobile phones, tablets, personal computers, smart screens, wearable head-mounted devices (for example, virtual reality head-mounted devices and augmented reality head-mounted devices), and other devices, wherein the wearable head-mounted devices may be augmented reality (AR) glasses and virtual reality (VR) glasses.

The present application embodiment is described by taking a mobile phone as an example of an electronic device. FIG2 shows a schematic diagram of the structure of a mobile phone 100 .

As shown in Figure 2, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen (touch screen) 194, and a subscriber identification module (SIM) card interface 195, etc.

It is to be understood that the structure shown in this embodiment does not constitute a specific limitation on the mobile phone 100. In other embodiments, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. In an embodiment of the present application, the processor 110 obtains the IR image and depth image of the user, and determines the user's gaze information on the display screen (touch screen) 194 of the mobile phone 100 based on the two images. Specifically, the NPU can obtain the user's IR image and depth image, and determine the user's gaze information on the display screen (touch screen) 194 of the mobile phone 100 based on the two images.

The controller may be the nerve center and command center of the mobile phone 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (SIM) interface, and/or a universal serial bus (USB) interface, etc.

It is understandable that the interface connection relationship between the modules shown in this embodiment is only a schematic illustration and does not constitute a structural limitation on the mobile phone 100. In other embodiments, the mobile phone 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 is charging the battery 142, it can also power the electronic device through the power management module 141.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. In some embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

The wireless communication function of the mobile phone 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor. In some embodiments, the antenna 1 of the mobile phone 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the mobile phone 100 can communicate with the network and other devices through wireless communication technology.

Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 150 can provide solutions for wireless communications including 2G/3G/4G/5G, etc., applied to the mobile phone 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation.

The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 can be set in the same device as at least some modules of the processor 110.

The wireless communication module 160 can provide wireless communication solutions for application on the mobile phone 100, including WLAN (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), nearfield communication technology (NFC), infrared technology (IR), etc.

The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 may also receive a signal to be sent from the processor 110, modulate the signal, amplify the signal, and convert it into an electromagnetic wave for radiation via the antenna 2.

The mobile phone 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor. The ISP is used to process the data fed back by the camera 193. The camera 193 is used to capture static images or videos. In some embodiments, the mobile phone 100 may include 1 or N cameras 193, where N is a positive integer greater than or equal to 1. In the embodiment of the present application, after the mobile phone 100 starts the camera 193, real-time image data can be obtained.

Camera 193 can be a time of flight camera (TOF). TOF is a depth information measurement solution, which is mainly composed of an infrared light projector and a receiving module. The projector projects infrared light outward, and the infrared light is reflected after encountering the object to be measured and is received by the receiving module. By recording the time from the emission to the reception of the infrared light, the depth information of the illuminated object is calculated, and 3D modeling is completed. The TOF camera can obtain IR images and depth images. The TOF camera can be used to project IR light onto a person's face to obtain an IR image of the user's face. Figure 3 shows a schematic diagram of an application scenario for obtaining a user's face image. For example, as shown in Figure 3, the mobile phone 100 can start an infrared camera to take an IR image of the face. In addition, IR images and depth images can also be obtained through a 3D structured light camera, but are not limited to this.

The mobile phone 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In the embodiment of the present application, the gaze information may include the gaze point information or gaze area information of the user on the display screen 194 of the mobile phone 100 .

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

The internal memory 121 may be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. For example, in an embodiment of the present application, the processor 110 may execute the instructions stored in the internal memory 121, and the internal memory 121 may include a program storage area and a data storage area.

The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, a service preemption function, etc.), etc. The data storage area may store data created during the use of the mobile phone 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

The mobile phone 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

The button 190 includes a power button, a volume button, etc. The button 190 can be a mechanical button. It can also be a touch button. The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 195, or pulled out from the SIM card interface 195 to achieve contact and separation with the mobile phone 100. The mobile phone 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than or equal to 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.

The present application embodiment provides a method for estimating a gaze point, which can be applied to an electronic device (such as a mobile phone 100) having the above hardware structure. FIG4 shows a flow chart of a method for estimating a gaze point. As shown in FIG4, the method for estimating a gaze point provided by the present application embodiment may include the following steps:

401 : The mobile phone 100 collects the IR image and depth image of the user through the camera 193 .

Since the photographic effect of the IR image is not affected by ambient light. Therefore, even in a dark or overly bright scene, the IR image captured by the mobile phone 100 can also contain a clearer user facial image, and the user's eye area image (i.e., the first eye area image hereinafter) and the position information of the user's eye key points in the user's eye area image are extracted from the IR image. Since the contents of the user's IR image and the depth image are basically the same, and the positions of the same content in each image are basically the same, the three-dimensional features corresponding to the image for indicating the user's eye key points can be obtained from the depth image through the aforementioned position information of the user's eye key points in the user's eye area image. In addition, the mobile phone 100 can also extract two-dimensional features (i.e., the first gaze feature hereinafter) for indicating the user's eyes and gaze-related from the eye area image corresponding to the IR image. In this way, the mobile phone 100 can determine the user's gaze information on the display screen 194 based on the aforementioned two types of gaze feature information.

In some embodiments, the mobile phone 100 needs a trigger condition to collect the IR image and depth image of the user through the camera 193. The following examples introduce several ways to obtain the IR image and depth image under the trigger conditions:

Method 1:

Some applications can be set in the mobile phone 100, and the application can support the gaze point estimation method of the present application. Then, after starting the application, the mobile phone 100 can periodically collect infrared images and depth images of the user through the preset camera 193. Specifically, the mobile phone 100 receives a first operation, and the first operation is used to trigger the mobile phone 100 to start the preset application; in response to the first operation, after starting the preset application, the mobile phone 100 periodically collects infrared images and depth images of the user through the preset camera 193. The preset application can be an e-book application, a browser application, a news application, etc., but is not limited thereto.

Method 2:

The collaborative working mode may refer to a state in which the mobile phone 100 can determine the user's gaze point, determine the user's intention based on the user's gaze point, and perform corresponding operations based on the user's intention. For example, in a scenario where a user is reading an e-book, the mobile phone can determine the user's gaze point, determine whether the user has the intention to turn the page based on the user's gaze point, and if so, actively provide the user with a page turning operation. For another example, in a scenario where a web page is browsed, the mobile phone 100 can determine the user's gaze point, determine the content that the user is interested in based on the user's gaze point, and actively recommend the user the content that the user is interested in. If the mobile phone 100 is in a collaborative working mode, the mobile phone 100 can also periodically collect the user's infrared image and depth image through the preset camera 193, so that the mobile phone 100 can more accurately detect the user's gaze information on the display screen 194. Specifically, if the mobile phone 100 is in a preset collaborative working mode, the mobile phone 100 periodically collects the user's infrared image and depth image through the camera 193.

In some embodiments, the mobile phone 100 may enter a preset collaborative working mode in response to a user turning on a preset collaborative switch, and the preset collaborative switch may be configured in a setting interface of the mobile phone 100, or in a control center of the mobile phone 100, or in a preset application of the mobile phone 100.

Method 3:

Some applications are set in the mobile phone 100. These applications require the mobile phone 100 to start a collaborative working mode and the mobile phone 100 periodically collects infrared images and depth images of the user through the preset camera 193 when the conditions for starting the application exist.

Among them, the preset collaborative switch can be configured in the setting interface of the mobile phone 100, or the preset collaborative switch can be configured in the control center of the mobile phone 100, or the preset collaborative switch can be configured in the preset application of the mobile phone 100. The mobile phone 100 can enter the preset collaborative working mode in response to the user's turning on operation of any of the preset collaborative switches.

It should be noted that the mobile phone 100 can start the collaborative working mode before or after starting the application.

Method 4:

In some embodiments, the mobile phone 100 can support the gaze point estimation method of the present application when it is turned on. Then, as long as the mobile phone 100 detects that the user is looking at the display screen 194, the gaze point estimation method of the present solution can be executed, thereby increasing the duration for which the mobile phone 100 executes the gaze point estimation method of the present solution. In this way, in a scenario where the mobile phone 100 assists the user in working according to the user's gaze point, the duration for which the mobile phone 100 accurately assists the user in working can be increased, thereby improving the user experience. Specifically, when the mobile phone 100 detects that the user is looking at the display screen 194, the infrared image and the depth image are collected by the preset camera 193.

402: The mobile phone 100 determines the user's gaze information on the display screen 194 based on the user's IR image and depth image.

The gaze information may include the gaze point information, the gaze area information, or the gaze point information and the gaze area information of the user on the display screen 194 of the mobile phone 100, wherein the gaze point information may be the coordinates of the gaze point, and the gaze area information may be the identification of the gaze area. For example, as shown in FIG5 , the display screen 194 of the mobile phone 100 may be divided into a plurality of gaze areas: gaze area 1 to gaze area 8, and the gaze area information may be the identification of the gaze areas: numbers 1 to 8. The gaze point information may be the coordinates (x1, y2) of the gaze point of the user on the display screen 194 of the mobile phone 100.

The mobile phone 100 can determine the user's gaze information on the display screen 194 based on the user's IR image and depth image. Specifically, a specific implementation method is provided below. For example, FIG5 shows a flowchart of a gaze point estimation method. As shown in FIG5, the following steps 501 to 504 are included:

501: The mobile phone 100 identifies an IR image, obtains a first eye area image of a user and first position information of an eye key point of the user in the first eye area image; wherein the first position information may be coordinates of each pixel of the eye key point of the user in the IR image, and the coordinates may be x-axis coordinates and y-axis coordinates in a preset two-dimensional coordinate system.

It can be understood that the key points of the eyes may include the human eye and components around the human eye, such as eyebrows, eyelids, eyeballs, eyelashes, corners of the eyes, pupils, etc.

It is understood that the mobile phone 100 can identify the IR image through some target detection algorithms to obtain the first eye area image of the user and the eye key points of the user in the first eye area image. For example, the first eye area image E of the user obtained by the mobile phone 100 from the IR image as shown in FIG1 .

The preset two-dimensional coordinate system may be a two-dimensional coordinate system pre-set by the mobile phone 100. Thus, after obtaining the eye key points of the user in the first eye area image, the mobile phone 100 may obtain the coordinates of each pixel point of the eye key points of the user in the first eye area image based on the two-dimensional coordinate system, and these pixel point coordinates constitute the first position information.

502: The mobile phone 100 obtains a first gaze feature based on the first eye region image, wherein the first gaze feature is used to indicate a two-dimensional feature of the user's eyes related to gaze.

The eye area image contains a lot of feature information related to the user's eyes and gaze, so the mobile phone 100 can obtain features related to the user's eyes and gaze based on the eye area image. The user's eyes and gaze-related features may include information about the human eye and components around the human eye, such as eyebrows, eyelids, eyeballs, eyelashes, eye corners, pupils, etc. Specifically, the first gaze feature may include morphology and state information of the human eye. The morphological information of the human eye may be the shape and size of the human eye, such as the shape and size of the pupil, the type of eyelids, and the size of the eye contour. The state information of the human eye may be dynamic information of the human eye, for example, the state information of the human eye may be information such as the rotation state of the eyeball, the position of the eyeball, the degree of opening of the eye corners, and the direction of the pupil.

503: The mobile phone 100 obtains a second gaze feature from the depth image based on the first position information, wherein the second gaze feature is used to indicate a three-dimensional feature of a key point of an eye of the user.

Since the contents of the user's IR image and depth image are basically the same, and the positions of the same content in their respective images are basically the same, the mobile phone 100 can use the first position information obtained above to obtain a second gaze feature indicating the three-dimensional features of the user's eye key points from the depth image that is also located in the preset two-dimensional coordinate system.

The second gaze feature includes 3D information of the user's eyes, and the 3D information of the user's eyes includes three-dimensional coordinate information (x, y, z), wherein x represents the x-axis position of the user's eyes in a preset three-dimensional coordinate system, y represents the y-axis position of the user's eyes in the preset three-dimensional coordinate system, and z represents the distance between the user and the optical center of the camera 193 of the mobile phone 100. The preset three-dimensional coordinate system may be a three-dimensional coordinate system with the optical center of the camera 193 as the origin.

In some embodiments, the mobile phone 100 may obtain the second gaze feature from a depth image also located in a preset two-dimensional coordinate system based on the first position information.

In some embodiments, the preset three-dimensional coordinate system has a corresponding mapping relationship with the aforementioned preset two-dimensional coordinate system. The mobile phone 100 can obtain the second gaze feature in the depth image from the mapping relationship based on the first position information. 504: The mobile phone 100 determines the user's gaze information on the display screen 194 based on the first gaze feature and the second gaze feature.

Since the first gaze feature is a two-dimensional feature for indicating the user's eyes and gaze-related features, and the second gaze feature is a three-dimensional feature for indicating the user's eye key points, the two-dimensional features of the user's eyes and gaze-related features and the three-dimensional features of the user's eye key points can predict the direction of the user's eyes, so the mobile phone 100 can first determine the direction of the user's eyes according to the first gaze feature and the second gaze feature, and then determine the user's gaze information on the display screen 194 based on the direction of the user's eyes.

Specifically, the process includes the following steps 5041-5042:

5041: The mobile phone 100 obtains a first eye posture parameter according to the first gaze feature; and obtains a second eye posture parameter according to the second gaze feature; wherein the eye posture parameter includes the sight direction information of the user's eyes, and the sight direction information of the user's eyes includes the rotation angle, pitch angle and heading angle of the user's eyes in a preset three-dimensional coordinate system; the preset three-dimensional coordinate system can be a three-dimensional coordinate system with the optical center of the camera 193 as the origin.

5042: The mobile phone 100 determines the user's gaze information on the display screen based on the first eye posture parameter and the second eye posture parameter.

In some embodiments, there is a preset correspondence between the eye posture parameters and the user's gaze information on the display screen 194. The mobile phone 100 can use the first eye posture parameters and the second eye posture parameters to determine the user's gaze information on the display screen 194 from the preset correspondence.

It can be understood that the model can have a powerful and fast prediction function. By using the human eye gaze information estimation model to obtain the user's gaze information on the display screen, the estimation speed of the gaze information can be improved, the user experience can be improved, and in the scenario where the electronic device assists the user in working according to the user's gaze point, the efficiency of the electronic device in assisting the user in working can be improved, and the user experience can be improved. Therefore, in some embodiments, the function of step 402 can be integrated into the human eye gaze information estimation model. In this way, the mobile phone 100 can input the user's IR image and depth image into the trained human eye gaze information estimation model, and use the human eye gaze information estimation model to determine the user's gaze information on the display screen 194 of the mobile phone 100.

For example, Fig. 6 shows a calculation schematic diagram of a human eye gaze information estimation model. As shown in Fig. 6, the human eye gaze information estimation model includes an eye region image determination module 1, a first gaze feature determination module 2, a second gaze feature determination module 3 and a gaze information output module 4.

The eye area image determination module 1 has the function of executing the content of the above step 501 .

The first gaze feature determination module 2 has the function of executing the content of the above step 502 .

The second gaze feature determination module 3 has the function of executing the content of the above step 503 .

The gaze information output module 4 has the function of executing the content of the above step 504. The gaze information output module 4 may be a model with a classification function, and may include a fully connected layer.

It can be understood that in the initial establishment stage of the human eye gaze information estimation model, the following steps 1 and 2 can be performed to obtain multiple training samples for training the above human eye gaze information estimation model. Step 1: Collect the user's IR image and depth map. Step 2: The user's real gaze information on the display screen 194 corresponding to the user's IR image and depth map.

It can be understood that the above-mentioned training samples are used to train the human eye gaze information estimation model in the stage of initially establishing the human eye gaze information estimation model. Therefore, the human eye gaze information estimation model after multiple sample trainings can be equipped with the user's IR image and depth map to obtain the user's gaze information on the display screen 194 corresponding to the user's IR image and depth map. Moreover, the more times the sample is trained, the higher the accuracy of the user's gaze information on the display screen 194 obtained by the human eye gaze information estimation model. Therefore, in the embodiment of the present application, the human eye gaze information estimation model pre-configured in the mobile phone 100 can be an AI model that has been trained with a large number of samples.

Exemplarily, the above-mentioned AI model can be a convolutional neural network. For example, the above-mentioned AI model can be a model structure based on a convolutional layer, a pooling layer, and a fully connected layer. The embodiments of the present application are not limited to this.

If the aforementioned human eye gaze information estimation model can output gaze information including gaze point information and gaze area information. Then the aforementioned human eye gaze information estimation model outputs the gaze area through a classification method, and improves the accuracy of gaze point estimation through a joint multi-task (classification and regression) learning method. Compared with existing solutions in the industry, the algorithm of this solution is more robust and has richer usage scenarios. In summary, even if the mobile phone 100 shoots the user in an environment where the scene is too dark or too bright, the mobile phone 100 can also obtain the user's eye geometric features, and then use the depth image to accurately determine the user's gaze point, so that the mobile phone 100 successfully assists the user in his work, thereby improving the reliability of the mobile phone 100 in assisting the user in his work.

In some cases, the user's head may not be still for a certain period of time, and may turn, look up, or lower the head, etc. In this way, the user's eye image may not exist in the IR image and depth image of the user taken by the mobile phone 100. If the user's eye image does not exist in the IR image and depth image of the user taken by the mobile phone 100, the mobile phone 100 will consume a certain amount of energy and time to execute this step. In order to save the energy and time consumed by the mobile phone 100 to execute this step, the mobile phone 100 may also first determine whether the user's eye image exists in the user's IR image. If so, execute this step. If not, re-execute step 401, that is, collect the user's IR image and depth image again through the camera 193.

The mobile phone 100 can use the user's gaze information on the display screen 194 of the mobile phone 100 to complete the collaborative work between the mobile phone 100 and the user. For example, FIG7 shows a schematic diagram of the interface of the mobile phone 100. As shown in FIG7, there is an e-book application in the mobile phone 100. After the user clicks the e-book application icon A, the e-book is read. In the process of the user reading the e-book, the mobile phone 100 can determine the user's gaze point and determine whether the user has the intention to turn the page based on the user's gaze point. For example, FIG8 shows a schematic diagram of the interface of an e-book application. As shown in FIG8, the mobile phone 100 can determine whether the user's gaze point is on the go to previous page button B1 or the go to next page button B2. If it is on the go to next page button B2, it actively provides the user with a turn to the next page operation. In this way, the user's intelligent experience of the collaborative work between the mobile phone 100 and the user can be improved.

The gaze point estimation method provided in the embodiments of the present application can be applied to human-computer interaction on smartphones, tablets, smart screens, and AR/VR glasses.

Existing gaze point estimation methods only estimate gaze points through RGB images, without considering the impact of images taken at different depths and postures on gaze points, resulting in reduced accuracy of gaze point estimation. However, this application is based on IR images, and the mobile phone 100 can also perform gaze point estimation in dark light scenes.

The existing human eye gaze information estimation model uses the extracted face image and face mesh as input. If the user wears a mask, face extraction may fail, and thus the gaze point estimation cannot be performed. However, this application is only based on eye features, and wearing a mask does not affect feature extraction, so it can cope with the scenario where the user wears a mask.

Another embodiment of the present application provides an electronic device, which includes: a memory and one or more processors. The memory is coupled to the processor. The memory also stores a computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device can execute the various functions or steps executed by the mobile phone 100 in the above method embodiment. The structure of the electronic device can refer to the structure of the mobile phone 100 shown in Figure 2.

An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the mobile phone 100 in the above-mentioned method embodiment.

The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the mobile phone 100 in the above method embodiment. The computer may be the above electronic device (such as the mobile phone 100).

The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and/or storage elements), at least one input device and at least one output device.

Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable storage media. Therefore, a machine-readable storage medium may include any mechanism for storing or disseminating information in a machine (e.g., computer) readable form, including but not limited to, a floppy disk, an optical disk, an optical disk, a read-only memory (CD-ROMs), a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for disseminating information (e.g., a carrier wave, an infrared signal digital signal, etc.) based on the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable storage media include any type of machine-readable storage media suitable for storing or propagating electronic instructions or information in a form readable by a machine (eg, a computer).

In the accompanying drawings, some structural or method features may be shown in a specific arrangement and/or order. However, it should be understood that such a specific arrangement and/or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and/or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

It should be noted that the units/modules mentioned in the various device embodiments of the present application are all logical units/modules. Physically, a logical unit/module can be a physical unit/module, or a part of a physical unit/module, or can be implemented as a combination of multiple physical units/modules. The physical implementation method of these logical units/modules themselves is not the most important. The combination of functions implemented by these logical units/modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units/modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units/modules in the above-mentioned device embodiments.

It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims (11)

1. The method for estimating the gaze point is characterized in that the method is applied to electronic equipment, the electronic equipment comprises a preset camera and a display screen, and the method comprises the following steps:

acquiring an infrared image and a depth image of a user through the preset camera;

Identifying an infrared image to obtain a first eye region image of a user and first position information of eye key points of the user in the first eye region image;

Obtaining a first fixation feature based on the first eye region image, wherein the first fixation feature is used for indicating two-dimensional features of eyes of a user related to fixation;

Obtaining a second fixation feature from the depth image based on the first position information, wherein the second fixation feature is used for indicating three-dimensional features of eye key points of a user;

acquiring first eye pose parameters according to the first gazing characteristics;

Acquiring a second human eye pose parameter according to the second gazing feature; the first human eye pose parameter and the second human eye pose parameter comprise sight line orientation information of eyes of a user, and the sight line orientation information of the eyes of the user comprises a rotation angle, a pitch angle and a course angle of the eyes of the user under a preset three-dimensional coordinate system; the preset three-dimensional coordinate system takes the optical center of the preset camera as an origin;

and determining the gazing information of the user on the display screen based on the first human eye pose parameter and the second human eye pose parameter.

2. The method of claim 1, wherein there is a preset correspondence between the first and second human eye pose parameters and gaze information of the user on the display screen;

the determining, based on the first human eye pose parameter and the second human eye pose parameter, gaze information of the user on the display screen includes:

and determining the gazing information of the user on the display screen according to the preset corresponding relation by utilizing the first human eye pose parameter and the second human eye pose parameter.

3. The method according to claim 1 or 2, wherein prior to the deriving a first gaze feature based on the first eye region image, the method further comprises:

and determining that the infrared image comprises a human eye image.

4. A method according to any one of claims 1-3, wherein the capturing of the infrared image and the depth image of the user by the preset camera comprises:

receiving a first operation, wherein the first operation is used for triggering the electronic equipment to start a preset application;

And responding to the first operation, and periodically acquiring the infrared image and the depth image of the user through the preset camera after the preset application is started.

5. A method according to any one of claims 1-3, wherein the capturing of the infrared image and the depth image of the user by the preset camera comprises:

And if the electronic equipment is in a preset cooperative working mode, periodically acquiring the infrared image and the depth image of the user through the preset camera.

6. The method of claim 5, wherein prior to the capturing the infrared image and the depth image of the user by the preset camera, the method further comprises:

responding to the starting operation of a user on a preset cooperative switch, and entering a preset cooperative working mode;

the preset cooperative switch is configured in a setting interface of the electronic equipment, or is configured in a control center of the electronic equipment, or is configured in a preset application of the electronic equipment.

7. The method according to any one of claims 1-6, wherein the capturing, by the preset camera, an infrared image and a depth image of a user comprises:

And when the user is detected to watch the display screen, acquiring the infrared image and the depth image through the preset camera.

8. The method according to any one of claims 1-7, wherein the preset camera is a TOF camera or a 3D structured light camera.

9. The method for estimating the gaze point is characterized in that the method is applied to electronic equipment, the electronic equipment comprises a preset camera and a display screen, and the method comprises the following steps:

acquiring an infrared image and a depth image of a user through the preset camera;

Taking the infrared image and the depth image as input, and operating a human eye gazing information estimation model to obtain gazing information of the user on the display screen;

wherein, the human eye gazing information estimation model is used for:

Identifying an infrared image to obtain a first eye region image of a user and first position information of eye key points of the user in the first eye region image;

Obtaining a first fixation feature based on the first eye region image, wherein the first fixation feature is used for indicating two-dimensional features of eyes of a user related to fixation;

Obtaining a second fixation feature from the depth image based on the first position information, wherein the second fixation feature is used for indicating three-dimensional features of eye key points of a user;

acquiring first eye pose parameters according to the first gazing characteristics;

Acquiring a second human eye pose parameter according to the second gazing feature; the first human eye pose parameter and the second human eye pose parameter comprise sight line orientation information of eyes of a user, and the sight line orientation information of the eyes of the user comprises a rotation angle, a pitch angle and a course angle of the eyes of the user under a preset three-dimensional coordinate system; the preset three-dimensional coordinate system takes the optical center of the preset camera as an origin;

and determining the gazing information of the user on the display screen based on the first human eye pose parameter and the second human eye pose parameter.

10. An electronic device comprising a memory and one or more processors; the memory is used for storing code instructions; the processor is configured to execute the code instructions to cause the electronic device to perform the method of any of claims 1-9.

11. A computer readable storage medium comprising computer instructions which, when run on an electronic device, cause the electronic device to perform the method of any of claims 1-9.