CN103885592B - Method and device for displaying information on screen - Google Patents

Abstract

The embodiment of the present invention discloses a method and device for displaying information on a screen, which relates to the technical field of intelligent terminals, and can determine the area that the user is currently focusing on on the screen by tracking the focus of the user's eyeballs on the screen, and the Data is displayed on the area of interest. The method of the present invention includes: using an eye tracking sensor to determine the area currently on the screen that is focused by the eye of the user operating the electronic terminal; and displaying target data on the determined area. Wherein, for an electronic terminal, the electronic terminal at least includes an eye tracking sensor and a screen, and the eye tracking sensor is used to track eye movement and determine the current focus position of the eye. The invention is suitable for operating electronic equipment without fingers.

Description

A method and device for displaying information on a screen

technical field

The present invention relates to the technical field of intelligent terminals, in particular to a method and device for displaying information on a screen.

Background technique

Intelligent terminals such as smartphones and PADs have become important tools for people to obtain information in their daily lives. At present, the operation mode of smart terminals on the market has been developed from keyboard operation to touch operation, so that users can complete operations as simple as possible when holding a smart terminal, especially when reading web pages, short messages, and emails. The user only needs to perform a simple sliding operation to trigger the smart terminal to display the information to be read on the screen.

However, the current smart terminal still needs to rely on the user to operate by hand. Users need to call out the information they want to read through finger touch operations, and display it on the screen before reading. During the reading process, users can perform operations such as turning pages, marking, and adjusting display windows through finger touch operations. In many cases, when the user is often unable to free his hands, for example: when the user is brushing teeth, playing games, etc. when the user's hands are busy, if the user wants to read the information, he needs to stop other actions at hand and operate the smart terminal with his fingers. Thereby, the information you want to read is called out and displayed on the screen. Therefore, in the process of actually using the smart terminal, the user often delays other things at hand due to the need to free up hands to operate the smart terminal.

Contents of the invention

Embodiments of the present invention provide a method and device for displaying information on the screen, which can determine the area that the user is currently focusing on on the screen by tracking the focus of the user's eyeballs on the screen, and display the information on the area that the user is focusing on. data.

In order to achieve the above object, embodiments of the present invention adopt the following technical solutions:

In a first aspect, an embodiment of the present invention provides a method for displaying information on a screen, which is used in an electronic terminal, the electronic terminal at least includes an eye tracking sensor and a screen, and the eye tracking sensor is used to track eye movement , and determine the position where the current focus of the eyeball is located; the method includes:

Using the eye-tracking sensor to determine an area currently on the screen that is focused by the eyes of the user operating the electronic terminal;

The target data is displayed on the determined area.

In the first aspect, an embodiment of the present invention provides a device for displaying information on a screen, which is used in an electronic terminal, the electronic terminal at least includes an eye tracking sensor and a screen, and the eye tracking sensor is used to track eye movement , and determine the position of the current focus of the eyeball; the device includes:

A positioning module, configured to use the eye tracking sensor to determine an area currently on the screen that is focused by the eyes of the user operating the electronic terminal;

The display module is used for displaying target data on the determined area.

The method and device for displaying information on the screen provided by the embodiments of the present invention can use the eye tracking sensor to determine the area currently on the screen that is focused by the eye of the user operating the electronic terminal; The target data is displayed on the area. Compared with the solution in the prior art that requires the user to display data on the screen through finger manipulation, the embodiment of the present invention determines the area that the user is currently focusing on on the screen by tracking the focus of the user's eyeballs on the screen, and Display data on the area of interest. In the process of using the electronic device, especially when the user needs to display the data required by the user on the screen, the user does not need to free up his hands for operation, so as to avoid delaying other things at hand.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

FIG. 1 is a flow chart of a method for displaying information on a screen provided by an embodiment of the present invention;

FIG. 2 is a flowchart of another method for displaying information on a screen provided by an embodiment of the present invention;

Figure 2a is a schematic diagram of a specific example provided by the embodiment of the present invention;

FIG. 3 is a flowchart of another method for displaying information on a screen provided by an embodiment of the present invention;

FIG. 4 is a flowchart of another method for displaying information on a screen provided by an embodiment of the present invention;

Fig. 4a is a schematic diagram of another specific example provided by the embodiment of the present invention;

5a and 5b are schematic diagrams of yet another specific example provided by the embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a device for displaying information on a screen according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of another device for displaying information on a screen according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of another device for displaying information on a screen according to an embodiment of the present invention.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. As used in the embodiments of the present invention and the appended claims, the singular forms "a", "said" and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present invention to describe various modules, units, connection ports and identification information, these modules, units, connection ports and identification information should not be limited to these terms. These terms are only used to distinguish modules, units, connection ports and identification information from each other. For example, without departing from the scope of the embodiments of the present invention, a first connection port may also be called a second connection port, and similarly, a second connection port may also be called a first connection port.

Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrases "if determined" or "if detected (the stated condition or event)" could be interpreted as "when determined" or "in response to the determination" or "when detected (the stated condition or event) )" or "in response to detection of (stated condition or event)"

An embodiment of the present invention provides a method for displaying information on a screen, as shown in FIG. 1 , including:

101. Using the eye tracking sensor, determine an area currently on the screen that is focused by an eyeball of a user operating the electronic terminal.

The embodiment of the present invention can be applied to an electronic terminal, which at least includes an eye tracking sensor and a screen, wherein the eye tracking sensor is used to track eye movement and determine the current focus position of the eye. The screen is used to display short messages, interface windows of various applications, text/picture information and other data.

102. Display target data on the determined area.

It can be understood that the target data mentioned in the embodiments of the present invention is one or a combination of short messages to be displayed on the screen, interface windows of various applications, text/picture information and other data.

The method for displaying information on the screen provided by the embodiment of the present invention can use the eye tracking sensor to determine the currently focused area on the screen that is focused by the eyes of the user operating the electronic terminal. The target data is displayed on the determined area. Compared with the solution in the prior art that requires the user to display data on the screen through finger manipulation, the embodiment of the present invention determines the area that the user is currently focusing on on the screen by tracking the focus of the user's eyeballs on the screen, and Display data on the area of interest. In the process of using the electronic device, especially when the user needs to display the data required by the user on the screen, the user does not need to free up his hands for operation, so as to avoid delaying other things at hand.

On the basis of the method flow shown in Figure 1, the embodiment of the present invention also provides a solution as shown in Figure 2:

201. Using the eye tracking sensor, determine an area currently on the screen that is focused by an eye of a user operating the electronic terminal.

202. Detect whether the area focused by the eyeball is in the first working state.

Wherein, the screen is divided into at least two areas. Wherein, one region has at least two working states, and the power consumption of one region in the first working state is lower than the power consumption of one region in the second working state. Specifically, the first working state may include states such as sleeping, dimming the brightness, and turning off the backlight; the second working state is a state in which the screen operates normally and displays data. When the screen is normally displaying data, it is often necessary to perform processes such as turning on the backlight; increasing the screen brightness according to the ambient brightness or system settings; Power consumption in low brightness, off backlight, etc.

In this embodiment, as shown in Figure 2a, the screen of the electronic device can be divided into multiple non-overlapping areas, and the split-screen technology can be used to control the working status of each area, for example, when the user's eyeball When the focus is in area 4, the electronic device can switch areas 1, 2, 3, 5, and 6 to the sleep state, and turn off the backlight of areas 1, 2, 3, 5, and 6; or the electronic device can set areas 1, 2, Adjust the brightness of 3, 5, and 6 to the minimum, and turn off the backlight of areas 1, 2, 3, 5, and 6. Therefore, only the area 4 is kept in a working state that can be used for displaying data at any time.

203. If it is in the first working state, switch the focus area of the eyeball to the second working state.

If it is in the second working state, go to step 204 .

204. Display target data on the determined area.

As shown in FIG. 3, after 102 or 204, this embodiment of the present invention may also include the following solutions:

301. Use the eye tracking sensor to track the focus of the eye of the user of the electronic terminal to move from the first area to the second area. At this time, the first area is in the second working state, so The second area is in the first working state.

Wherein, the area focused by the eyeball is the first area, and the electronic device displays the target data in the first area.

302. Switch the second area to the second working state, and display the target data in the second area.

303. Switch the first region to the first working state.

In order to further save the power consumption of the screen, before displaying the target data on the determined area, the electronic device can also take measures to reduce power consumption, such as sleeping, dimming the brightness, and turning off the backlight, for some areas on the screen. The area where the power consumption is measured is mainly the area outside the area where the eyeballs are focused. Therefore, the power consumption of the screen can be reduced when the user operates the electronic device through eyeball rotation.

Further, in this embodiment, different areas may be divided on the screen of the electronic device, and one area divided on the screen corresponds to at least one application program. When the user's eyes focus on an area, the electronic device can automatically run the application program corresponding to this area. As shown in Figure 4, the specific implementation of 102 or 204 may include:

401. Determine an application program corresponding to an area focused by eyeballs of a user operating the electronic terminal.

For example: as shown in Figure 4a, the screen of the electronic device is divided into 4 areas, and the corresponding relationship between each area and the application program is set in the electronic device. Among them, area 1 corresponds to the monitoring program used to monitor the operating status of the electronic device ; Area 2 corresponds to the time reporting program; Area 3 corresponds to the message reading program for displaying short messages; Area 4 corresponds to the prompt program, which is used when the electronic device receives update information, or when the electronic device is running anti-virus software and other programs And when an alarm message occurs, a prompt message for indicating that an update message has occurred or an alarm message has occurred is displayed.

402. Run the corresponding application program.

In this embodiment, there are many ways to run the application program corresponding to an area. For example, as shown in Figure 4a, when the focus of the eyeball is located in area 1, the signal strength indicator and the remaining battery indicator can be displayed on area 1. and other icons; when the focus of the eyeball is in area 2, the display interface of the timekeeping program can be displayed on area 2, and the current time can be displayed in the display interface of the timekeeping program; when the focus of the eyeball is in area 3, and there is When the text message is unread, the specific content of the unread text message can be displayed in area 3, such as the entire text or the first 10 text of the unread text message; when the focus of the eyeball is in area 4, a prompt message can be displayed in area 4 .

It should be noted that when the electronic device runs an application corresponding to an area and needs to display the corresponding content of the application, it can display the corresponding content of the application in this area; or it can display the area where the focus of the eyeball is or displayed within a certain range on the screen centered on the focus of the eyeball, without being limited by the size of the area; or all areas on the screen can be switched to the second working state, and through the entire screen to display the corresponding content of the application.

It should be further explained that if the electronic device displays the corresponding content of the application program in an area, and the focus of the eyeball on the screen moves, the electronic device can synchronously move the corresponding content of the application program to the eyeball on the screen. The position or area to which the focus of the user has recently moved, for example: at the first time, the focus of the eyeball on the screen is located in area 1, area 1 corresponds to the application program for reading short messages, and the area 1 of the electronic device displays the Read the short message; later, because the user turns the eyeball, the focus of the eyeball on the screen moves, and moves to area 2 at a second time, then the electronic device can transfer and display the unread short message originally displayed on area 1 on zone 2.

Different areas are divided on the screen of the electronic device, and one area divided on the screen corresponds to at least one application program. The number of application programs that can be invoked by the user through the eyeball rotation can be increased, so that the user can operate the electronic device through the eyeball in a richer and more diverse manner.

Furthermore, since the user's gaze changes on the screen at any time during actual use, the focus of the eyeball on the screen will continue to move. If a time period is limited, a period of focus of the eyeball on the screen can be obtained. movement track.

Further, the embodiments of the present invention may also include the following solutions:

501a. Using the eye tracking sensor, track the movement of the focus of the user's eye to a first action triggering area.

Wherein, at least one action triggering area is divided on the screen. The action triggering area corresponds to at least one operation action.

502a. Execute an operation action corresponding to the first action trigger area.

For example: as shown in Figure 5a, four action-triggered areas are divided along the four sides of the screen of the electronic device, and direction arrows representing different directions are displayed on the four action-triggered areas, wherein the action The operation action corresponding to trigger area 1 is page up; the operation action corresponding to action trigger area 2 is page down; the operation action corresponding to action trigger area 3 is page left; the operation action corresponding to action trigger area 4 is page up Turn the page right. In addition, an area 5 is divided in the center of the screen of the electronic device, and the area 5 corresponds to the web browsing program. At this time, since the focus of the user's eyeballs is located on the area 5, the electronic device opens the web browsing program and displays the content of the web browsing program on area 5. interface for users to browse the web. When the user needs to turn down the page, the user only needs to turn the eyeball and move the focus to the action trigger area 2, and then the electronic device performs a page-down action for the interface of the web browsing program.

or,

501b. Use the eye tracking sensor to track a moving track of the focus of the user's eye within a specified period of time. Wherein, the electronic terminal stores at least one correspondence between movement tracks and operation actions.

502b. Execute an operation action corresponding to the movement track of the user's eye focus.

For example, as shown in FIG. 5b, the screen of the electronic device is divided into 9 areas, and the corresponding relationship between the "Z"-shaped moving track and the screen unlocking action is set in the electronic terminal. When the user needs to unlock the screen, he can turn his eyes and move the focus in the order of Area 1→Area 2→Area 3→Area 5→Area 7→Area 8→Area 9, so that the moving track of the focus is in a "Z" shape , when the electronic device detects that the moving track of the focus is in a "Z" shape and corresponds to an action of unlocking the screen, the screen is unlocked.

Furthermore, in this embodiment, in the process of determining the operation action by using the eye tracking sensor to track the displacement of the focus of the eyeball, the user's eye closing times and closing time can also be used as a parameter for determining the operation action, for example : When the eye tracking sensor determines that the focus of the eyeball stays on area 1, where a "page turning" icon is displayed on area 1, and area 1 corresponds to the operation action of "turning pages", the user blinks once, Then the electronic device can regard the blinking of the user as a click operation, and determine that the user has clicked the "page turning" icon on the area 1, and turn the page of the web page content currently displayed on the screen. Likewise, rapid multiple blinks by the user can act as a double-tap.

It should be noted that, what 501a-502a provides is to firstly determine the area where the focus of the eyeball stays on the screen, and then determine the operation action according to the determined area. 501b-502b provide an operation action determined by tracking the movement trajectory of the eye focus. Therefore, the execution timing of 501a-502a and 501b-502b is not limited, and it can be executed simultaneously with 101-103 or 201-204, or it can be executed at any time during the user's use of the electronic device, so as to realize the rotation of the eyeballs on the electronic device Operate, thereby replacing the operation mode of operating on the electronic device by fingers.

An embodiment of the present invention provides a device 60 for displaying information on a screen. The device 60 can be used in an electronic terminal. The electronic terminal includes at least an eye tracking sensor and a screen, and the eye tracking sensor is used to track eye movement. , and determine the position of the current focus of the eyeball; as shown in Figure 6, the device 60 includes:

The positioning module 61 is configured to use the eye tracking sensor to determine the area currently on the screen that is focused by the eyes of the user operating the electronic terminal;

A display module 62, configured to display target data on the determined area.

Further, as shown in FIG. 7, the device 60 also includes:

A detection module 63, configured to detect whether the eyeball-focused area is in the first working state before displaying the target data on the determined area;

Wherein, the screen is divided into at least two areas; wherein, one area has at least two working states, and the power consumption when the one area is in the first working state is lower than the power consumption when the one area is in the second working state ;

The switching module 64 is configured to switch the focused area of the eyeball to the second working state if it is in the first working state.

Optionally, the display module 62 may include:

A query unit 621, configured to determine the application program corresponding to the area focused by the eyeballs of the user operating the electronic terminal;

Wherein an area divided on the screen corresponds to at least one application program;

The running unit 622 is configured to run the corresponding application program.

Further, the detection module 63 is also used to detect that the first area or the second area is in the first working state or the second working state; wherein, the area focused by the eyeball is the first working state. an area, the electronic device displays target data in the first area;

The switching module 64 is further configured to use the eye tracking sensor to track the focus of the user's eyeballs of the electronic terminal when moving from the first area to the second area. At this time, the first area is in the In the second working state, the second area is in the first working state; switching the second area to the second working state, and displaying the target data in the second area; and The first area is switched to the first working state.

Further optionally, as shown in FIG. 8, the device 60 may also include:

The first action recognition module 65 is configured to use the eye tracking sensor to track the movement of the focus of the user's eyeballs to the first action trigger area; wherein, at least one action trigger area is divided on the screen; the action trigger area Corresponding to at least one operational action;

The first trigger module 66 is configured to execute the operation action corresponding to the first action trigger area;

And/or, the second action recognition module 67 is configured to use the eye tracking sensor to track the moving track of the focus of the user's eye within a specified period of time; wherein, at least one movement track is stored in the electronic terminal Correspondence between trajectory and operation action;

The second trigger module 68 is configured to perform an operation action corresponding to the moving track of the user's eye focus.

The device for displaying information on the screen provided by the embodiment of the present invention can use the eye tracking sensor to determine the area currently on the screen that is focused by the eye of the user operating the electronic terminal; in the determined area The target data is displayed on the Compared with the solution in the prior art that requires the user to display data on the screen through finger manipulation, the embodiment of the present invention determines the area that the user is currently focusing on on the screen by tracking the focus of the user's eyeballs on the screen, and Display data on the area of interest. In the process of using the electronic device, especially when the user needs to display the data required by the user on the screen, the user does not need to free up his hands for operation, so as to avoid delaying other things at hand.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiment.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the programs can be stored in a computer-readable storage medium. During execution, it may include the processes of the embodiments of the above-mentioned methods. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) or a random access memory (Random Access Memory, RAM), and the like.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. All should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (8)

1. The method for displaying information on the screen is characterized by being used for an electronic terminal, wherein the electronic terminal at least comprises an eyeball tracking sensor and the screen, and the eyeball tracking sensor is used for tracking the activity condition of eyeballs and determining the position of the current focus of the eyeballs; the screen is divided into at least two areas; the power consumption of one area in the first working state is lower than that of the area in the second working state; the method comprises the following steps:

determining, by the eye tracking sensor, an area currently on the screen focused by eyes of a user operating the electronic terminal;

displaying the target data on the determined area;

before displaying the target data on the determined area, the method further comprises:

detecting whether an area focused by the eyeballs is in a first working state or not;

if the eyeball is in the first working state, switching the area focused by the eyeball to the second working state;

the method further comprises the following steps:

tracking, by the eye tracking sensor, movement of a focal point of an eye of the user to a first action trigger area; wherein at least one action triggering area is divided on the screen; the action trigger area corresponds to at least one operation action, and the operation action comprises direction operations in different directions;

and executing the operation action corresponding to the first action trigger area.

2. A method for displaying information on a screen according to claim 1, wherein an area divided on the screen corresponds to at least one application; the displaying the target data on the determined area includes:

determining an application program corresponding to an area focused by eyeballs of a user operating the electronic terminal;

and running the corresponding application program.

3. The method of claim 1, wherein the area on which the eyeball is focused is a first area, and the electronic terminal displays target data in the first area;

the method further comprises the following steps:

when the focus of the eyeballs of the user of the electronic terminal is tracked to move from the first area to the second area by using the eyeball tracking sensor, the first area is in the second working state at the moment, and the second area is in the first working state;

switching the second area to the second working state, and displaying the target data in the second area;

and switching the first area to the first working state.

4. The method of displaying information on a screen of claim 1, further comprising:

tracking, with the eye tracking sensor, a movement trajectory of a focal point of an eye of the user over a specified time period; the electronic terminal stores the corresponding relation between at least one moving track and operation actions;

and executing the operation action corresponding to the movement track of the focus of the eyeball of the user.

5. The device for displaying information on the screen is characterized by being used for an electronic terminal, wherein the electronic terminal at least comprises an eyeball tracking sensor and a screen, and the eyeball tracking sensor is used for tracking the activity condition of eyeballs and determining the position of the current focus of the eyeballs; the device comprises:

the positioning module is used for determining an area focused by eyeballs of a user operating the electronic terminal on the screen by using the eyeball tracking sensor;

a display module for displaying the target data on the determined area;

the screen is divided into at least two areas; the power consumption of one area in the first working state is lower than that of the area in the second working state;

the device further comprises:

the detection module is used for detecting whether the area focused by the eyeballs is in a first working state or not before the target data is displayed on the determined area;

the switching module is used for switching the area focused by the eyeballs to the second working state if the switching module is in the first working state;

the device further comprises:

a first action recognition module, configured to track, by using the eyeball tracking sensor, that a focal point of an eyeball of the user moves to a first action trigger area; wherein at least one action triggering area is divided on the screen; the action trigger area corresponds to at least one operation action, and the operation action comprises direction operations in different directions;

and the first trigger module is used for executing the operation action corresponding to the first action trigger area.

6. An apparatus for displaying information on a screen according to claim 5, wherein an area divided on the screen corresponds to at least one application; the display module includes:

the query unit is used for determining an application program corresponding to an area focused by eyeballs of a user operating the electronic terminal;

and the running unit is used for running the corresponding application program.

7. The apparatus for displaying information on a screen according to claim 5, wherein the area on which the eyeball is focused is a first area, and the electronic terminal displays the target data in the first area;

the switching module is further configured to track, by using the eyeball tracking sensor, when a focus of an eyeball of a user of the electronic terminal moves from the first area to a second area, at this time, the first area is in the second working state, and the second area is in the first working state; switching the second area to the second working state, and displaying the target data in the second area; and switching the first area to the first working state.

8. An apparatus for displaying information on a screen as recited in claim 5, further comprising:

a second motion recognition module for tracking a movement trajectory of a focal point of an eyeball of the user within a specified time period by using the eyeball tracking sensor; the electronic terminal stores the corresponding relation between at least one moving track and operation actions;

and the second trigger module is used for executing the operation action corresponding to the movement track of the focus of the eyeball of the user.