JP7157444B2 - Information terminal with camera and program - Google Patents

Description

The present invention relates to an information terminal with a camera attached to the head and a program.

A user interface is known that controls a device based on the orientation of the head including the face and the orientation of the line of sight. For example, Patent Document 1 describes a head-mounted display (1) having a display (11) on which an icon (described as a pointer in the document) is displayed. This is a head-mounted device equipped with a camera (12) for detecting the direction of the user's line of sight and an acceleration sensor (13) for detecting the movement of the user's head. This device is a pointing system that allows the user to click or drag as in a personal computer operation by shaking the head up and down or left and right while placing the eye on the icon. Since the relative positions of the user's eyeballs and the display (1) remain unchanged, the eyeballs do not rotate when the head is shaken unless the positions of the icons in the display (1) change. In other words, there is no line of sight change. When the user removes his/her eye from the icon, shaking his/her head is not regarded as a click action, and malfunctions are prevented from unintentionally shaking his/her head.

Further, Patent Document 2 discloses that although the device is not a head-mounted device, the user maintains the viewpoint (described as line of sight in the document) at the icon (30) of the display section (20) of the display placed on the desk. This document describes a gesture recognition device that determines that an icon (30) has been selected by tilting the head back and forth or returning it (referred to as a nodding gesture in the document). The camera 10 provided above the display unit (20) detects the user's line of sight and the orientation of the face in front of the head. In this device, the positional relationship between the user's head and the display (20) is not relatively fixed. Therefore, if the user makes a nodding gesture while keeping the viewpoint at the icon (30), the user's eyeballs rotate, and the direction of the head and the direction of the line of sight change in a direction that does not match. Become. In this way, Patent Document 3 also describes a method of expressing an intention by a motion that does not match the direction of the line of sight and the direction of the face.

Further, Patent Document 4 describes an imaging device that controls the zoom operation of the camera by the user's gaze at a predetermined mark on the finder screen (102). This is because indexes (201, 202) indicating the wide side and the tele side of the zoom operation are displayed in the corners of the finder screen (102), and when the user gazes at the display position of the desired zoom ratio index for a predetermined time, , the zoom operation of the imaging apparatus is performed. A mark called a zoom rate index is provided in a corner of the viewfinder screen (102), which is a place where the user does not normally gaze, and is used in the user interface to prevent unintentional malfunction by the user.

Further, Patent Document 5 describes an image forming apparatus having a step (103) of performing a process of enlarging the display of a portion of a touch panel type operation display unit (106) to which the user's line of sight is directed. This is configured such that when the user's finger is detected near the display unit (106), the enlarged display process is not performed. In this way, it is possible to prevent impairing the operability due to unintentional enlargement of the display when the user is performing a finger touch operation.

Further, Patent Document 6 describes a navigation device that enlarges and displays a screen by gazing at a plurality of predetermined points. In this way, the screen is enlarged and displayed only when a plurality of points are gazed at, thereby preventing enlargement and display against the user's intention. In this way, techniques for operating devices based on the line of sight, face, and head movements have been disclosed. These are accompanied by a configuration for not operating against the user's intention.

However, when trying to apply these configurations to the zoom operation of a head-mounted camera, the following problems arise. For example, if the configuration of Patent Document 1 is applied, the user must shake the head up and down or left and right with a certain amount of acceleration while gazing at the icon displayed on the viewfinder of the camera or an equivalent screen. When the head is shaken, the camera attached to the head is also shaken, resulting in a problem that the object to be photographed cannot be captured well.

Further, with the configurations described in Patent Documents 2 and 3, when the user's viewpoint is placed on the object to be photographed, the direction of the head is out of the object to be photographed. Since the direction of the head-mounted camera is the same as the direction of the head, the camera is also out of the shooting target, making it difficult for the user to operate the zoom while capturing the target on the screen.

By the way, it is preferable that the center of the viewfinder of the camera or the screen corresponding thereto coincides with the center of the camera. This is because this makes it easier for the user to grasp the center and range of the image captured by the camera. Based on this, if the configuration described in Patent Document 4 is applied to a camera, it would be unnatural to have to look at the zoom icon at the corner of the screen for the zoom operation while capturing the object to be photographed in the center of the screen. A task that forces action arises. If the zoom icon is displayed in the vicinity of the center of the screen to avoid this, the user often gazes at the vicinity of the center of the screen in the first place, so there is a problem that the malfunction of zooming against the user's intention is likely to occur. .

In addition, if the configuration of Patent Document 5 is applied to the zoom operation of a head-mounted camera, in addition to eyeball and head movements, hand movements such as fingers are also required, and both hands are said to be free. The problem arises that the original merit cannot be realized. Further, if the configuration of Patent Document 6 is applied to the same zoom operation, it is necessary to sequentially watch a plurality of icons in the screen, which causes a problem that the procedures are complicated and time consuming.

JP 2006-243784 A JP-A-2000-163196 JP 2007-94619 A JP-A-8-46846 JP 2018-18437 A JP 2018-77105 A

SUMMARY OF THE INVENTION It is an object of the present invention to provide an information terminal with a camera and a program capable of performing a zoom operation while gazing at an object to be photographed, which is an operation by an eye movement and a head movement while preventing malfunction. be.

As an embodiment for solving the above problems, a view camera (11) attached to the face of the user, a posture change detection section (13) for detecting a posture change of the view camera (11), and the user's eyeballs. a camera-equipped information terminal having an eyeball camera (14) for photographing, a line-of-sight detection unit (15) for detecting the line of sight from the image of the eyeball camera (14), and a display unit (16) visible to the user can do. Then, when the direction of the change in the posture of the field-of-view camera (11) detected by the posture change detection unit (13) and the line of sight detection unit (15) and the direction of change in the line of sight are mutually offset, the display unit A gaze icon (IC) is displayed near the center of (16), and while the user's viewpoint remains at the gaze icon (IC), the zoom ratio of the view camera (11) is controlled to be large or small. .

The embodiment of the present invention can provide an information terminal with a camera and a program that can perform a zoom operation while gazing at an object to be photographed while preventing malfunction, and being an operation by an eyeball movement and a head movement. .

FIG. 1 is a diagram of a camera-equipped information terminal according to a first embodiment of the present invention. FIG. 2 is a diagram for explaining a photographing system and a display system of the camera-equipped information terminal according to the first embodiment of the present invention. FIG. 3 is a diagram showing an equivalent field of view camera and an equivalent display section of the camera-equipped information terminal according to the first embodiment of the present invention. FIG. 4 is a diagram for explaining the relationship between the user's actions and the display on the display unit. FIG. 5 is a diagram for explaining a gaze icon. FIG. 6 is a diagram showing the vicinity of the center of the display section. FIG. 7 is a block diagram showing the configuration of the camera-equipped information terminal according to the first embodiment of the present invention. FIG. 8 is a processing flow diagram of the camera-equipped information terminal according to the first embodiment of the present invention. FIG. 9 is a diagram for explaining an operation in which the line of sight of the user changes downward. FIG. 10 is a diagram showing the operation of decreasing the zoom rate. FIG. 11 is a processing flow diagram of the camera-equipped information terminal according to the second embodiment of the present invention. FIG. 12 is a diagram for explaining a method of detecting a line of sight from an eyeball camera image. 13A and 13B are diagrams for explaining the operation of the posture change detection unit according to the first and second embodiments. FIG. FIG. 14 is a diagram showing the configuration of an information system according to the third embodiment based on the present invention. FIG. 15 is a diagram showing the relationship between the user's neutral viewpoint, the center of the display unit, and the center of the field-of-view camera (fourth embodiment).

[First embodiment information terminal with camera]
A camera-equipped information terminal 1 according to a first embodiment of the present invention will be described with reference to FIG. FIG. 1A is a diagram showing a state in which a user wears the camera-equipped information terminal 1. FIG. (A2) shows the right eye side of the user. FIGS. 1B1 and 1B2 are diagrams showing a part of the camera-equipped information terminal 1 transparently. As shown in FIG. 1, each component of the camera-equipped information terminal 1 is attached to the user's head by a frame 1F. The right eye side is covered with a display section cover 16A that covers up to the front of the right ear. Inside the display unit cover 16A, a field-of-view camera 11 for photographing the foreground of the user, an eyeball camera 14 for photographing the right eye of the user, and an image captured by the field-of-view camera 11 are arranged so that the user can visually recognize them. A display unit 16 for displaying is provided. In addition, in FIG. 1B1, the display section cover 16A is shown transparently. The interior contains prisms and mirrors necessary for display and photography, but the details will be omitted here since they will be described with reference to FIG. The field-of-view camera 11 shown in FIG. 1(A2) is also mounted in such a way that it cannot be visually recognized from the outside.

A control box 2A is also attached to the frame 1F. It contains a computing unit 21, a communication unit 22, and a memory 23, which will be described later in detail. The calculator 21 performs posture change detection processing for detecting a posture change of the view camera 11 and line-of-sight detection processing for detecting the user's line of sight from the image acquired by the eyeball camera 14 .

[Shooting system and display system]
The imaging system and the display system of the camera-equipped information terminal 1 will be described with reference to FIG. The optical path LP1 is guided to the field-of-view camera 11 by the imaging prism 16C, and moving images are captured. The photographing prism 16C has a substantially triangular prism shape, and is shown as a triangle in FIG. The second of the three sides is made up of mirrors that reflect light for all angles of incidence. Therefore, the light path LP1 carrying the user's foreground does not enter the right eye R-Eye directly. The user's foreground is photographed by the field-of-view camera 11 and displayed on the display unit 16 almost in real time. The user can visually recognize his or her foreground through the display unit 16 . An optical path LP2 of light emitted from the display section 16 is guided to the right eye R-Eye by the inner wall of the display section cover 16A, the mirror 16B, and the viewing prism 16D. Among the surfaces of the viewing prism 16D, the surface facing the photographing prism 16C is composed of a mirror that reflects light with respect to all incident angles. In this way, an image acquired by the field-of-view camera 11 mounted on the user's head is visually recognized by the user via the display unit 16 mounted on the user's head.

[Equivalent view camera and equivalent display part]
In the camera-equipped information terminal 1 of the first embodiment described with reference to FIGS. 1 and 2, the field-of-view camera 11 and the display section 16 are incorporated in the display section cover 16A and are not visible from the outside. On the other hand, in the camera-equipped information terminal 1 shown in FIGS. 3A and 3B, the field-of-view camera 11 and the display unit 16 are exposed to the outside. 1 and 2 and that shown in FIG. 3 are equivalent in the following respects.

First, the field-of-view camera 11 of the camera-equipped information terminal 1 according to FIGS. 1 and 2 can acquire the foreground of the user's head. The orientation of the field-of-view camera 11 also changes as the orientation of the user's head changes. From this point of view, the field-of-view camera 11 shown in FIGS. 3A and 3B and the field-of-view camera 11 of FIGS. 1 and 2 are equivalent. Also, the display unit 16 of the camera-equipped information terminal 1 shown in FIGS. Even if the posture of the user's head changes, the relative positional relationship between the display unit 16 and the user's right eye R-Eye remains unchanged. In this respect the display 16 shown in FIGS. 3A and 3B is equivalent to the display 16 according to FIGS. Therefore, hereinafter, description will be made based on the field-of-view camera 11 and the display section 16 shown in FIG.

By the way, in FIG. 3, the display unit 16 seems to block the front of the field-of-view camera 11 . However, the following description will be made on the assumption that the display unit 16 is out of the field of view of the field-of-view camera 11 . This is because the camera-equipped information terminal 1 shown in FIGS. 1 and 2 avoids interference such as mutual shielding of the other by the prism and the mirror. That is, in FIGS. 1 to 3, the field-of-view camera 11 is mounted on the face side of the user in the direction in which the tip of the user's nose is directed, so that the foreground of the user can be photographed. Also, the display unit 16 is mounted on the head so that the user can visually recognize the image acquired by the visual field camera 16 . Further, as a premise, it is assumed that the center of the image acquired by the field-of-view camera 11 is adjusted so as to be displayed at the center of the display unit 16 . Then, when the field-of-view camera 11 performs a zoom operation, the acquired image is enlarged or reduced with the center as the base point.

[User Action]
Next, with reference to FIG. 4, important user actions in the embodiment will be described. FIG. 4 is a diagram for explaining changes in the image acquired by the field-of-view camera 11 and changes in the image displayed on the display unit 16 in accordance with the user's actions. 4(A1), (B1) and (C1) are diagrams showing the actions of the user. On the other hand, FIGS. 4A 2 , 4 B 2 and 4 C 2 are images displayed on the display unit 16 and also images captured by the field-of-view camera 11 .

FIG. 4A1 shows a state in which the user gazes at a small white square displayed in the center of the display unit 16 and gazes at it. That is, the user's line of sight VD is directed toward the center of the display unit 16 . Further, as described above, since the center of the image acquired by the visual field camera 11 has been adjusted so that it is displayed at the center of the display unit 16, the facing direction CD of the visual field camera 11 matches the direction of the user's line of sight VD. ing. Here, for convenience of explanation, the point at which the line of sight VD of the user intersects with the surface of the display unit 16 is called a viewpoint VP.

FIG. 4A2 shows an image displayed on the display unit 16 at this time. The user is gazing at the small white square in the center of display 16 . A line segment that is horizontal to the screen of the display unit 16 and passes through the center is displayed as a camera line CL. A line segment passing through the viewpoint VP and horizontal to the screen of the display unit 16 is displayed as a viewpoint line VL. In this case, the camera line CL and the viewpoint line VL match.

[Offset action]
FIG. 4(B1) shows a state in which the user's head is turned slightly downward while keeping the viewpoint at the small square which is the previous gaze point. This is also the state in which the user has slightly pulled the chin. The view camera 11 and the display unit 16 mounted on the user's head are tilted at the same angle as the head. As a result, as shown in FIG. 4B2, the image displayed on the display unit 16 moves slightly upward on the screen. At this time, there is a difference between the facing direction CD of the field-of-view camera 11 and the direction of the user's line of sight VD. A difference also occurs between the camera line CL and the viewpoint line VL, which is represented as two separate line segments. As in the example shown here, when the orientation of the head is changed while the viewpoint VP remains in a specific portion of the foreground, the line of sight changes in a direction that cancels out the orientation change of the head. Here, such an action that causes a change in the line of sight in the direction opposite to the change in the posture of the head is also referred to as a counteracting action.

[Gaze icon]
FIG. 4(C1) and FIG. 4(B1) are exactly the same posture. There is no change in user behavior. Therefore, the image displayed in FIG. 4(C2) matches the image displayed in FIG. 4(B2). However, in FIG. 4C2, the gaze icon IC is displayed in the center of the display section 16. FIG.

FIG. 5 is a diagram for explaining the gaze icon IC displayed on the display unit 16. As shown in FIG. FIG. 5(A1) shows a user with the same head posture as FIG. 4(C1). However, the viewpoint VP remains at the gaze icon IC. As a result, the direction CD of the field-of-view camera 11 and the direction of the user's line of sight VD match. On the screen of the display unit 16 shown in FIG. 5A2, the camera line CL and the viewpoint line VL are aligned. However, the viewpoint line VL is shifted below the small white square [see FIG. 4(A2)] where the viewpoint VP was originally staying.

Here, it is easy to align the point of view VP with the small white square where the point of view VP stays at the first, while keeping the point of view VP at the fixation icon IC. What is necessary is to stick out the chin slightly while maintaining the fixation of the viewpoint on the gaze icon IC. FIG. 5B1 shows the state immediately after this operation is performed. By this operation, as shown in FIG. 5B2, the image on the display unit 16 moves slightly downward, and the camera line CL, viewpoint line VL, and gaze icon IC overlap the small white square. The posture of the user at this time matches FIG. 4 (A1). A series of actions from FIG. 4 (A1) to (B1), (C1), and further through FIG. 5 (A1) to (B1) are usually not actions performed unconsciously by the user. It is an action that you do not do unless you are conscious of it. Although this is an unnatural operation, it can be easily performed by displaying the gaze icon IC on the display unit 16 from FIG. 4(C2) to FIG. 5(A2) and (B2). 5C and 5D show that the center of the display unit 16, that is, the focus icon IC is displayed with the zoom ratio gradually increased.

In principle, the gaze icon IC is displayed in the center of the display unit 16 . However, it is not limited to this, and may have a spread from the center of the display section 16 . For example, as shown in FIG. 6A, it is an area that spreads to some extent from the center of the display section 16 . This is called a center area 161, and the gaze icon IC is displayed in this area. The central region 161 is shown enlarged for clarity. As shown in FIG. 6B, the line of sight that is separated by two degrees upward from the state in which the user is gazing at DS0, which is the center of the display unit 16, is defined as VDP, and the line of sight is separated by two degrees downward. be VDM. At this time, the center region 161 is defined by the intersection of the line of sight VDP and the display unit 16 and the intersection of the line of sight VDM and the display unit 16 . The horizontal range of the display unit 16 is defined by a range in which the line of sight is tilted 2 degrees to the left from the center DS0 of the display unit 16 and a range in which the line of sight is tilted 2 degrees to the right. This range is simply expressed as within a viewing angle of plus or minus 2 degrees from the center of the display section 16 . The gaze icon IC may be superimposed on the captured image of the view camera 11 and displayed in the central region 161 within the viewing angle range of plus or minus 2 degrees.

[Configuration block diagram]
Next, referring to FIG. 7, the configuration of the first embodiment will be described with a block diagram. The camera-equipped information terminal 1 according to the first embodiment has a field-of-view camera 11 , an eyeball camera 14 , a display section 16 , a calculator 21 , a memory 23 and a communication section 22 . The arithmetic unit 21 may have the posture change detection unit 13, the zoom control unit 12, the line-of-sight detection unit 15, and the display control unit 17 as electronic circuits. Alternatively, it may be implemented by processing based on a program that executes equivalent processing. The program may be incorporated in the computing unit 21 or expanded in the memory 23 . The computing unit 21 controls the field-of-view camera 11, the eye camera 14, the display unit 16, the memory 23, and the communication unit 22, and these devices constitute the camera-equipped information terminal 1 according to the first embodiment. ing.

For example, the arithmetic unit 21 controls the zoom function of the field-of-view camera 11 by using the zoom control unit 12 or a program corresponding thereto to enlarge or reduce the image acquired by the field-of-view camera 11 . This image is displayed on the display section 16 under the control of the display control section 17 . Also, this image may be stored in the memory 23 or transmitted to another device via the communication section 22 . In addition, posture change detection unit 13 can detect a posture change of view camera 11 from a change in the image of view camera 11 . The detection method will be described later. The change in the orientation of the field-of-view camera 11 also means that the orientation of the head of the user of the camera-equipped information terminal 1 changes.

In addition to the above functions, the display control unit 17 performs control to display or erase the gaze icon IC in the center of the display unit 16 or the center area 161 when a predetermined condition is satisfied. The line-of-sight detection unit 15 processes the eyeball image of the user acquired by the eyeball camera 14 to detect the line of sight of the user. Detecting the line of sight means detecting the direction of the line of sight or the coordinates of the point where the line of sight intersects the surface of the screen of the display unit 16 as the viewpoint VP.

[Processing flow]
Next, with reference to FIG. 8, the processing flow of the camera-equipped information terminal 1 of the first embodiment will be described. FIG. 8A is the processing flow for detecting the user's motion described with reference to FIGS. 4 and 5. FIG. It can be executed by the calculator 21 . While the user is using the camera-equipped information terminal 1, this process is repeatedly executed. This process detects a change in the user's line of sight and a change in the posture of the field-of-view camera 11. If the directions of the two changes are in opposite directions, that is, if the changes cancel each other out, the gaze icon IC is displayed on the display unit 16. Display at or near the center of Then, if the user stays the viewpoint VP on the attention icon IC, the display is maintained, and if the user does not stay the viewpoint VP, the display of the attention icon IC is deleted.

First, the computing unit 21 detects a change in posture of the view camera 11 (step STP1), and detects the line of sight of the user from the image acquired by the eyeball camera 14 (step STP2). Either step STP1 or step STP2 may be executed first. Next, branch based on condition 1. Condition 1 is 'true' if the user's viewpoint VP is outside the central area 161 . As for the condition of the line-of-sight displacement from the vicinity of the center, the condition 1 may be a larger displacement. However, hereinafter, the fact that the viewpoint VP is outside the center region 161 is also expressed as "change in line of sight". This is because the user's viewpoint VP is normally within the center region 161 and the viewpoint VP deviates from the center region 161 in accordance with some action of the user. If it is not 'true', that is, if it is 'false', the process ends, and if it is 'true', it branches to the subsequent process (step STP3).

Then, condition 2 is judged. Condition 2 is a determination based on a comparison between a change in line of sight, that is, the direction in which the viewpoint VP deviates from the center region 161, and a change in the posture of the view camera 11, that is, the direction in which the user's head posture changes. In FIGS. 4B1 and 4C1, the viewpoint VP deviates upward from the central region 161. In FIG. On the other hand, when the user tilts the head forward, the posture of the head-mounted field-of-view camera 11 changes downward. In such a case, the line-of-sight change is in a direction that cancels out the camera posture change, so the judgment of condition 2 is 'true'. If the change in camera posture is not detected, or if the direction of camera posture change is not opposite to the direction of line-of-sight change, it is determined as 'false' rather than cancellation. Based on the determination of condition 2, if 'false', the process is terminated, and if 'true', the process branches to the subsequent process (step STP4).

Subsequently, the gaze icon IC is displayed in the center area 161 . This is displayed superimposed on the image acquired by the field-of-view camera 11 . In addition, it sets a gaze flag FG, which will be explained later. After the gaze icon IC is displayed, there is a predetermined time interval before proceeding to the next process (step STP5). This is to ensure that the gaze icon IC is displayed in the center area 161 and that the user maintains the viewpoint VP there. Although it is desirable that this predetermined time is within 1 second, setting exceeding 1 second is not excluded.

Next, the line of sight is detected again (step STP6), and condition 3 is determined. In the judgment of condition 3, if the user's viewpoint VP is out of the gaze icon IC, it is 'false', and if it stays, it is 'true'. It branches based on the determination of condition 3 (step STP7). If 'false', reset the gaze flag FG, erase the gaze icon IC, and terminate the process (step STP9). On the other hand, in the case of 'true', the process returns to the line-of-sight detection of step STP6 (step STP7). Here, the gaze flag FG is a binary reference flag such as ON and OFF that can be referred to in other processing routines.

To simply express the above processing, when the change in the line of sight and the movement of the field-of-view camera 11 mounted on the user's head are offset, the gaze icon IC is displayed in the center area 161 . When the user stops the viewpoint VP here, the display of the gaze icon IC is maintained, but when the user does not stop the viewpoint VP, the gaze icon IC is erased. Further, while the display of the gaze icon IC is maintained, the gaze flag FG remains set.

Here, the computing unit 21 is a computing unit that operates in multithread. That is, multiple processing routines are executed in parallel. The configuration of the calculator 21 may be either a single processor or a multiprocessor. In the case of a single processor, multiple processing routines are executed in a time-sharing manner, apparently in parallel. In the case of multiprocessors, it is assumed that multiple processors access common memory and memory access contention is appropriately regulated.

The processing shown in FIG. 8(B) is performed in parallel with the processing described with reference to (A). Furthermore, it is possible to refer to the gaze flag FG that is output in the process described above. Here, condition 4 is first determined. In condition 4, if the gaze flag FG is set, it is 'true', and if it is reset, it is 'false'. If it is 'false', the process ends, and if it is 'true', it branches to the subsequent process (step STP10). Next, the zoom operation of the field-of-view camera 11 is performed via the zoom control section 12 (step STP11). Then, condition 4 is determined again (step STP10). By executing this process, while the gaze flag FG is set, that is, while the gaze icon IC is displayed in the center area 161, the zoom operation of the view camera 11 is continued.

According to the processing flow described above, the user of the camera-equipped information terminal according to the first embodiment can perform the zoom operation of the field-of-view camera 11 only by moving the head and the eyeballs. If the viewpoint VP continues to stay at the gaze icon IC displayed in the center region 161 near the center of the display unit 16, the gaze flag FG continues to be set and the zooming operation of the view camera 11 can be continued. Further, when the gaze icon IC is displayed in the center region 161, if the line-of-sight detection unit 15 detects that the gaze VP point is out of the gaze icon IC, the gaze icon IC is displayed at step STP9. It is erased, and at the same time, the gaze flag FG is also reset. As a result, the camera zoom operation shown in FIG. 8B also ends.

The chin pulling action of the user has already been described with reference to FIGS. Along with the chin-pulling motion, the face on the front of the head tilted downward, while the line of sight changed upward, and the two directions changed in opposite directions to offset each other. A contrasting action is the chin thrusting action by the user. This is explained with reference to FIG. FIG. 9 (A1) shows a normal, natural state, in which the face direction and the line-of-sight direction match. The direction CD of the field-of-view camera 11 mounted on the front of the head coincides with the line-of-sight direction CD of the user whose viewpoint VP is set near the center of the display section 16 . FIG. 9A 2 is an image acquired by the field-of-view camera 11 and is also an image displayed on the display unit 16 . In this image, the camera line CL and the viewpoint line VL match near the center of the display section 16 . The user is gazing at a small white square appearing near the center of display 16 .

FIG. 9B1 shows a state in which the user sticks out his chin while maintaining his gaze on the small white square displayed near the center of the display unit 16, and as a result, his face is turned slightly upward. there is At this time, the orientation of the field-of-view camera 11 changes upward, and the viewpoint VP changes downward. The image acquired by the field-of-view camera 11 at this time and displayed on the display unit 16 is shown in (B2). Compared with the image of (A2), it has moved slightly downward. Of course, the camera line CL remains positioned at the center of the display section 16 . On the other hand, the line of sight VL is moving downwards along with the small white squares. At this time, since the direction of change in posture of the view camera 11 and the direction of change in the user's line of sight are opposite to each other and offset each other, the gaze icon IC is displayed in the central region 161 of the display unit 16 [FIG. 9 ( C2)].

FIG. 10(A1) shows a state in which the user keeps the viewpoint VP staying on the gaze icon IC and superimposes the small white square that was initially gazed on the gaze icon IC. (A2) is the display screen of the display unit 16 at this time. In FIGS. 10B and 10C, a zoom operation is performed to reduce the zoom rate around the center of the display unit 16, that is, the gaze icon IC, so that the object gradually becomes smaller and the field of view is gradually expanded. It shows how it is going.

4 and 5 are examples in which the gaze icon IC is displayed when the direction of change in the line of sight is upward, and in FIGS. This is an example in which an icon IC is displayed. In addition, it is possible to create a case where the change in line of sight changes to either the left or right direction, and the sight camera 11 changes its posture in a direction that offsets this change. In the second embodiment described below, in the operation of displaying the gaze icon IC, when the change in line of sight is upward, the view camera 11 is caused to perform an enlargement zoom operation, and when the change in line of sight is downward, This causes the visual field camera 11 to perform a reduction zoom operation. Here, the expansion zoom is a zoom operation in which an object becomes larger and the field of view becomes narrower. The reduction zoom is a zoom operation in which the object becomes smaller and the field of view becomes wider.

[Second embodiment information terminal with camera]
A processing flow of the camera-equipped information terminal 2 according to the second embodiment of the present invention will be described with reference to FIG. The configuration represented by the block diagram of the camera-equipped information terminal 2 is the same as that of the camera-equipped information terminal 1 of the first embodiment. First, FIG. 11A is referred to. The processing from step STP1 to step STP4 is the same as that of the camera-equipped information terminal 1 of the first embodiment. In other words, if the direction of line-of-sight change and the direction of camera attitude change do not cancel each other out, the process ends. If there is a canceling relationship, the process branches to step STP12.

Condition 5 is that the line-of-sight change detected in the previous step is an upward change. The change in line of sight upward is a change that occurs when the user pulls the chin and turns the head slightly downward. If condition 5 is judged and the change in line of sight is not upward, it is determined as 'false' and the process branches to step STP14. If the line-of-sight change is upward, it is determined as 'true' and the process branches to step STP13 (step STP12). At step STP13, a gaze U flag, which will be explained later, is set. This is a 1-bit flag that is set when the line-of-sight change was upward, i.e. when the user pulled his chin. This flag is referred to in the processing routine described later with reference to FIG. 11(B). At step STP14, a gaze D flag is set. This is a 1-bit flag that is set when the line-of-sight change was downward, i.e., when the user stuck out his chin. This flag is referred to in the processing routine described later with reference to FIG. 11(B). However, in the processing flow shown here, the fixation D flag is set for all line-of-sight changes other than upward, including the fact that the line-of-sight change was downward. Strictly speaking, if the gaze D flag is to be set only when the line-of-sight change is downward, it is necessary to branch step STP12 in three ways. Then, if the line-of-sight change does not correspond to any of the vertical directions, the processing may be terminated.

In the subsequent processing, the gaze icon IC is displayed in the center region 161 and maintained for a predetermined time (step STP5). This is to secure time until the user stays the viewpoint VP on the gaze icon IC. Next, the line of sight is detected again (step STP6), and condition 3 is determined. In the determination of condition 3, if the user's viewpoint VP does not stay at the gaze icon IC, it is 'false', and if it stays, it is 'true'. It branches based on the determination of condition 3 (step STP7). If 'false', reset the gaze U flag or gaze D flag, erase the gaze icon IC, and terminate the process (step STP15). On the other hand, in the case of 'true', the process returns to the line-of-sight detection of step STP6 (step STP7). Here, the attention U flag and the attention D flag are binary reference flags such as ON and OFF that can be referred to in other processing routines.

To simply express the above processing, when the change in the line of sight and the movement of the field-of-view camera 11 mounted on the user's head are offset, the gaze icon IC is displayed in the center area 161 . When the user stops the viewpoint VP here, the display of the gaze icon IC is maintained, but when the user does not stop the viewpoint VP, the gaze icon IC is erased. Further, while the display of the gaze icon IC is maintained, the gaze U flag or the gaze D flag remains set. When the user's motion is pulling the chin, the gaze U flag is set, and when the user's motion is the chin thrusting motion, the gaze D flag is set.

The processing shown in FIG. 11(B) is performed in parallel with the processing described with reference to (A). Further, it is possible to refer to the attention U flag and the attention D flag output in the processing described above. Here, condition 6 is first determined. In condition 6, it is 'true' if the gaze U flag is set, and 'false' if it is reset. If 'false', the process branches to step STP18, and if 'true', the process branches to step STP17 (step STP16). At step STP17, the field-of-view camera 11 is zoomed in and the process returns to step S16 (step STP17). At step STP18, condition 7 is judged. Condition 7 is 'true' if the gaze D flag is set, and 'false' if it is reset. If 'false', terminate the process. On the other hand, in the case of 'true', the reduction zoom operation of the view camera 11 is performed. Alternatively, it may be an operation to return to the maximum viewing range. After that, the process returns to step 16 (step STP19).

According to the processing flow described above, a change in posture of the field-of-view camera 11 is detected so that the user faces downward. Further, when it is detected that the viewpoint VP has changed upward from the center area 161 , the gaze icon IC is displayed within the center area 161 . Then, while the user stays at the gaze icon IC with the viewpoint VP, control can be performed to increase the zoom ratio of the field-of-view camera 11 .

Further, when a change in posture of the view camera 11 is detected so that the user faces upward and a change in the viewpoint VP from the central region 161 downward is detected, the gaze icon IC is displayed in the central region 161. Is displayed. Then, while the user stays at the gaze icon IC with the viewpoint VP, control can be performed to decrease the zoom ratio of the view camera 11 .

Further, when it is detected that the viewpoint VP is out of the gaze icon IC, the gaze icon IC is erased from the display section 16 . At the same time, the gaze U flag or the gaze D flag is also reset, so the operation of changing the zoom ratio of the field-of-view camera 11 ends.

Next, the line-of-sight detection unit 15 will be described with reference to FIG. The line-of-sight detection unit 15 detects the user's line of sight from the eyeball image acquired by the eyeball camera 14 . More specifically, the coordinates on the surface of the display unit 16 of the viewpoint VP, which is the place where the user's line of sight and the surface of the display unit 16 intersect, are detected. In the first and second embodiments, the position of the pupil of the user's right eye is detected from the eyeball image. Near-infrared illumination can also be applied to the right eye to stabilize pupil detection. In this case, the eyeball camera 14 uses a camera with high sensitivity in the near-infrared region.

Then, prior to line-of-sight detection, a calibration operation shown in FIGS. 12(A1) and (A2) is required. Calibration points P1, P2, P3, and P4 are displayed at predetermined positions of the display units 16 of the camera-equipped information terminals 1 and 2 of the first and second embodiments [FIG. 12 (A2) ]. A user wearing the information terminal follows these designated calibration points with his/her eyes, and the position of the pupil in the eyeball image at this time is recorded. The purpose of this is to associate the position of the user's viewpoint VP with the position of the pupil. If this correspondence is established, for example, when the pupil E1 is detected from the right-eye eyeball image shown in FIG. It is judged that there is Here, it should be noted that the surface (B1) of the display unit 16 visually recognized by the user and the pupil (B3) of the image from the eyeball camera are substantially opposite to each other, and thus the left and right are reversed. (B3) When the pupil is detected at a position that does not match any of the eyeways, the coordinates of the viewpoint VP are detected by moving in or out of the position of the calibration point eyeway in (B1). The above is the processing of the line-of-sight detection unit 15 .

Next, the posture change detection unit 13 will be described with reference to FIG. 13 . FIG. 13(A2) shows an image acquired by the field-of-view camera 11 mounted on the user's head when the camera is facing the direction CD1 [FIG. 13(A1)]. Here, when the user pulls the chin and tilts the head slightly downward, the view camera 11 is oriented toward CD2. An image acquired by the camera at this time is shown in FIG. 13 (B2). Here, if the optical flow is obtained from the difference between the images of (A2) and (B2), the posture change of the field-of-view camera 11 can be detected. Although the optical flow detection method is not limited, for example, (A2) creating a local template pattern from the image [T1, T2, T3 in FIG. 13 (C1)], (B2) pattern matching for the image Then, a matching area can be obtained as shown in (C2) [M1, M2, M3 in (C2)], and an optical flow is detected as a vector V2 from the position of T2 to the position of M2. The magnitude and direction of this vector are the magnitude and direction of the attitude change of the field-of-view camera 11 . Then, the direction opposite to the direction of this vector is the direction of change in posture of the field-of-view camera 11 . In this way, by processing the images before and after the change in camera posture, the change in camera posture and orientation can be detected. That is, the posture change detection unit 13 can be realized by the optical flow detection unit. The above is the processing of the posture change detection unit 13 . In order to detect changes in camera posture, an acceleration sensor or a gyro sensor may be attached to the user's head in addition to detecting optical flow from an acquired image.

The camera-equipped information terminals 1 and 2 according to the first and second embodiments are described above. With such a configuration, it is possible to perform a zoom operation of the camera while gazing at the object to be photographed while suppressing erroneous operations.

[Third Embodiment Information System]
Next, referring to FIG. 14, an information system 3 according to a third embodiment of the present invention will be described. The information system 3 of the third embodiment is composed of a head-mounted terminal 3A and a server 3B. A personal computer can be used as the server 3B. The images acquired by the field-of-view camera 11 provided in the head-mounted terminal 3A and the eyeball camera 14 are transmitted to the server 3B. The server 3B processes the image of the field-of-view camera 11 and detects the posture change of the field-of-view camera 11 . Further, the image of the eyeball camera 14 is processed to detect a line-of-sight change.

Then, when the server 3B detects that the change in the posture of the view camera 11 and the change in the line of sight obtained from the image of the eye camera 14 are in a direction that cancels each other, the server 3B instructs the head-mounted terminal 3A to gaze at the head-mounted terminal 3A. Send a command to display the icon IC. The head-mounted terminal 3A that has received the command displays the gaze icon IC in the center area 161 of the display unit 16 provided therein. After that, the server 3B obtains the line of sight from the image of the eyeball camera 14, and when it determines that the viewpoint VP, which is the intersection of the line of sight with the surface of the display unit 16, is outside the gaze icon IC, the head-mounted terminal 3A to delete the gaze icon IC. On the other hand, when the viewpoint VP stays at the gaze icon IC, the server 3B instructs the head-mounted terminal 3A to zoom the view camera 11 and increase or decrease the zoom rate. send. The head-mounted terminal 3A that has received the command controls the zoom ratio of the field-of-view camera 11 to be larger or smaller.

Based on the above operation, FIG. 14A is referred to. The configuration of the head-mounted terminal 3A is the same as the configuration of the first embodiment shown in FIG. 7 except that the functions of the posture change detection section and the line-of-sight detection section are removed. More specifically, the arithmetic unit 31A is the same as the arithmetic unit 21 except for the functions of the posture change detection section and the line-of-sight detection section. These processes are performed by the server 3B. With such a configuration, the load on the computing unit 31A is reduced more than that on the computing unit 21. FIG.

FIG. 14B shows the configuration of the server 3B. The server calculator 31B includes a line-of-sight detector 35, a posture change detector 33, a zoom operation commander 32, and a display control commander . These may be composed of dedicated electronic circuits, or may be realized as processing routines for processing equivalent functions by software. Posture change detection unit 33 detects an optical flow from the image of view camera 11 to detect a posture change of view camera 11 . If the head-mounted terminal 3A is equipped with an acceleration sensor or a gyro sensor, the processing may be changed to receive the output of these devices and detect this optical flow. The line-of-sight detection unit 35 detects line-of-sight and line-of-sight changes from the image of the eye camera 14 . The display control command unit 34 commands the head-mounted terminal 3A to display or erase the gaze icon IC on the display unit 16. FIG. The zoom operation command unit 32 sends a command to the zoom control unit 12 of the head-mounted terminal 3A to cause the field-of-view camera to perform a zoom operation. Communication between the head-mounted terminal 3A and the server 3B is performed via the communication section 22 and the server communication section 32. FIG. The above is the configuration of the information system 3 according to the third embodiment. With such a configuration, the load on the computing unit 31A of the head-mounted terminal 3A is lightened, and as a result, heat generation of the head-mounted terminal 3A can be suppressed, power consumption can be reduced, and weight reduction can be easily achieved. It has the effect of becoming

[Fourth embodiment]
A fourth embodiment according to the present invention will now be described with reference to FIG. This embodiment relates to the relationship between the center of the image acquired by the field-of-view camera 11 and the center of the display unit 16, and is an embodiment that can be added to all the above-described first to third embodiments. 15A shows the positional relationship between the center IM0 of the image acquired by the field-of-view camera 11 and the center DS0 of the display unit 16. FIG. An image 113 formed by the lens system 111 of the field-of-view camera 11 is converted into an electric signal by the image sensor 112 and transmitted to the display unit 16 . Then, this image is displayed on the display unit 16 as an image 16S. At this time, the center IM0 of the image 113 is displayed at the center DS0 of the display unit 16. FIG. This is based on the premise that the user usually puts the viewpoint at the center DS0 of the display unit 16. FIG. Since the center IM0 of the image 113 coincides with the center DS0 of the display section 16, when the field-of-view camera 11 performs a zoom operation, the display image 16S is enlarged or reduced centering on the portion shown in the center DS0 of the display section 16. FIG. Therefore, in the first to third embodiments, the gaze marker IC is displayed at the center of the display section 16. FIG.

However, there are individual differences between people, and the line of sight is not necessarily directed toward the center DS0 of the display unit 16 in a normal state. In addition, camera-equipped information terminals are not necessarily manufactured after being adjusted for each person, unlike individual eyeglasses. If so, it is assumed that the number of cases in which the viewpoint is placed at the center DS0 of the display unit 16 in the normal state is even less. Accordingly, the fourth embodiment is configured to display an image according to the position where each user usually places the viewpoint.

In the camera-equipped information terminal of the fourth embodiment, the coordinate values representing the position of the viewpoint VP of the user are acquired as samples at all times or at regular intervals, and stored in the memory 23 or the server memory 33 . By statistically processing this, the coordinates of the position where the user's viewpoints VP gather most or the coordinates of the position where the viewpoints VP gather on average are obtained. For statistical processing, an average value or a mode value can be taken. It is preferable to acquire samples except for the period during which the user displays the attention icon IC for zooming. This is because the period during which the user's viewpoint is guided to the gaze icon IC cannot be said to be the user's normal viewpoint position. By discarding the old sample and replacing it with the new sample, it follows the gradual change of the position of the user's normal viewpoint VP.

In the example of FIG. 15B, the coordinates of the most frequent value at which the viewpoint VP is placed are obtained by statistical processing. Here, the coordinate of the most frequent value at which the viewpoint VP is placed is called a neutral viewpoint position DS1. An image 16Q moved so that the center IM0 of the image 113 coincides with the neutral viewpoint position DS1 is displayed on the display unit 16. FIG. (C) is an enlarged representation of the main part for explanation. In the drawing, the neutral viewpoint position DS1 appears to move vertically on the paper with respect to the center position DS0 of the display unit 16, but in reality it can also move in a direction orthogonal to the paper. A center region 161 is also set around the neutral viewpoint position DS1.

In the fourth embodiment described above, the position corresponding to the average value or the mode value of the coordinate values representing the position of the viewpoint VP is detected and set as the neutral viewpoint position DS1, and the image 113 acquired by the field camera 11 at this neutral viewpoint position DS1. are displayed on the display unit 16 by matching the center IM0 of . Furthermore, a central region 161 is provided around the neutral viewpoint position DS1.

In addition, in order to implement these processes using a program, there is a step of detecting the position corresponding to the average value or the mode value of the coordinate values representing the position of the viewpoint VP and setting it as the neutral viewpoint position DS1. In the step of displaying the image captured by the field-of-view camera 11 and the gaze icon IC on the unit 16, the center of the image captured by the field-of-view camera 11 is aligned with the neutral viewpoint position DS1. Then, a center region 161 is defined with the neutral viewpoint position DS1 as the center, and processing is performed to display the gaze icon IC in this center region 161 . The series of processes described above may be executed by the arithmetic unit 21 or the arithmetic unit 31B. By doing so, it is possible to absorb the difference in the way of placing the viewpoint for each user and to perform natural operation for everyone. As described above, the position corresponding to the average value or the mode value of the coordinate values representing the position of the viewpoint VP is detected, and the image 16Q obtained by matching the center IM0 of the image 113 acquired by the field-of-view camera 11 to this position is displayed on the display unit 16. An information terminal with a camera was explained.

Note that in FIG. 15, the image 113 acquired by the field-of-view camera 11, the display image 16S on the display unit, and the corrected display image 16Q are represented like thick objects. However, this is used for convenience of explanation, and since these images are electric signals such as electric charges, they are not actually thick as shown in the figure.

1... information terminal with camera according to first embodiment, 11... field-of-view camera,
111... Lens system of field camera, 112... Image sensor of field camera,
113... Image acquired by view camera, 12... Zoom control unit,
13 Posture change detection unit 14 eyeball camera 15 line of sight detection unit
16... display section, 17... display control section, 1F... frame,
16A... display unit cover, 16B... mirror, 16C... imaging prism,
16D: Prism for visual recognition, 16S: Display image on the display unit,
16Q: display image after correction on the display unit, 161: central region,
2A... Control box, 21... Computing unit, 22... Communication unit,
23... memory, 2... information terminal with camera according to the second embodiment,
3... Information system of the third embodiment, 3A... Head-mounted terminal,
3B... server, 31A... calculator, 31B... server calculator,
32... Zoom operation command unit, 33... Posture change detection unit,
34... Display control command unit, 35... Line of sight detection unit, CL... Camera line,
VL... viewpoint line, E1, E2, E3, E4... detected pupils,
T1, T2, T3... templates, M1, M2, M3... matching regions,
IC: gaze icon, FG: gaze flag,
IM0: the center of the image acquired by the field-of-view camera, DS0: the center of the display unit,
DS1: Neutral viewpoint position, LP1, LP2: Optical path,
P1, P2, P3, P4... calibration points,
R-Eye ... the right eye of the user of the camera-equipped information terminal of the first embodiment,
VD: Direction and direction of the user's line of sight,
CD, CD1, CD2 -- direction and direction of the view camera,
VP: viewpoint, V1, V2, V3: optical flow vector

Claims (8)

a vision camera mounted on the face side of the user's head in the direction in which the tip of the user's nose points, and tilted by the same angle as the posture of the user's head and face changes ;
an eyeball camera mounted on the user's face side for photographing the user's eyes;
a display unit mounted on the head of the user so that the user can visually recognize the image, and displaying an image acquired by the field-of-view camera;
a posture change detection unit that processes information sent from a device worn on the head of the user and detects whether or not there is a posture change and the orientation of the field-of-view camera ;
The above calculated from the image acquired by the eyeball camera when the posture change detection unit detects a posture change of the field-of-view camera that accompanies the downward inclination of the face caused by pulling the chin of the user. When the direction of the change in the user's line of sight is the direction that offsets the change in the posture of the sight camera,
The gaze icon is configured to be displayed for a predetermined time in a central region within a viewing angle range of plus or minus 2 degrees from the center of the display unit,
When the viewpoint, which is the point where the surface of the display unit and the line of sight of the user intersect, stays on the gaze icon, the display of the gaze icon is maintained, and enlargement control is performed to increase the zoom rate of the view camera. An information terminal with a camera. The above calculated from the image acquired by the eyeball camera when the posture change detection unit detects a posture change of the field-of-view camera that accompanies an upward tilt of the face caused by the user sticking out the chin. The gaze icon is configured to be displayed for a predetermined period of time in the center region when the direction of change in the line of sight of the user is the direction that offsets the change in the posture of the field-of-view camera,
2. A camera-equipped information terminal according to claim 1, wherein when said viewpoint stays on said gaze icon, the display of said gaze icon is maintained, and reduction control is performed to reduce the zoom ratio of said view camera. Detecting a position corresponding to an average value or a mode value of coordinate values representing the position of the viewpoint on the surface of the display unit and calling it a neutral viewpoint position,
aligning the center of the image acquired by the field-of-view camera with the neutral viewpoint position and displaying the image on the display unit;
3. The camera-equipped information terminal according to claim 1, wherein said central region is provided around said neutral viewpoint position. Having a gyro sensor or an acceleration sensor mounted on the user's head and detecting a change in posture of the head and face ,
Based on the information sent from the gyro sensor or the acceleration sensor, the posture change detection unit detects the direction of the posture change of the head and face and the direction of the posture change of the viewing camera tilted by the same angle as the head. 4. The camera-equipped information terminal according to any one of claims 1, 2, or 3, which detects . 2. The orientation change detection unit detects an optical flow from a change in an image acquired by the sight camera , and detects the orientation of the orientation change of the sight camera from the direction of a vector detected as the optical flow. 4. The information terminal with a camera according to claim 2 or 3. A vision camera mounted on the user's face toward the direction of the user's nose tip and tilted by the same angle as the user's head and face posture changes, wherein the image of the vision camera is A gyro sensor or an acceleration mounted on the user's head that is tilted by the same angle as the field camera to detect the direction of the change in the posture of the field camera from the direction of the vector detected as the optical flow generated by the change an attitude change detection unit that detects the direction of attitude change of the viewing camera based on information sent from the sensor ;
a line-of-sight detection unit that receives an image acquired by an eyeball camera attached to the face of the user to photograph the eyes of the user and detects the line of sight of the user from the image;
a zoom operation command unit that sends a command to a zoom control unit of the field-of-view camera;
a display control command unit that is mounted on the user's head so that the user can visually recognize the image, and that sends a command to a display control unit of a display unit that displays an image acquired by the visual field camera. death,
When the posture change detection unit detects a change in posture of the field-of-view camera that accompanies a downward tilt of the face caused by pulling the chin of the user, the line of sight detected by the line-of-sight detection unit is detected by the field-of-view camera. If it is determined that it is changing in a direction that offsets the attitude change of
The display control command unit issues a command to the display control unit to display the gaze icon for a predetermined time in a central region within a viewing angle range of plus or minus 2 degrees from the center of the display unit,
If the viewpoint, which is the point where the line of sight detected by the line-of-sight detection unit intersects the surface of the display unit, stays on the gaze icon, the display control unit displays the gaze icon via the display control command unit. to maintain
Further, the camera-equipped information terminal server causes the zoom control section to perform enlargement control to increase the zoom ratio of the view camera through the zoom operation command section. a step of detecting a change in posture of a field-of-view camera mounted on the face side of the user in the direction in which the tip of the user's nose points;
a step of detecting the line of sight of the user from an image acquired by an eyeball camera provided in front of the user's face toward the user's eyeballs;
when the orientation of the orientation change of the visual field camera and the orientation of the visual line change detected in the step of detecting the orientation change of the visual field camera and the step of detecting the line of sight are mutually offset,
The center of a display unit that is worn on the user's head so that the user can visually recognize it and displays the image acquired by the visual field camera, or a viewing angle range of plus or minus 2 degrees from the center of the display unit causing the display unit to display a gaze icon superimposed on the image acquired by the field-of-view camera in a center area within the
a step of controlling the zoom ratio of the view camera while the viewpoint, which is the point where the line of sight of the user intersects the surface of the display unit, stays on the gaze icon. to
When a change in posture of the field-of-view camera caused by a downward tilt of the face caused by the user pulling the chin is detected, the direction of the change in the line of sight is a direction that offsets the change in the posture of the field-of-view camera. In some cases,
A program for causing, in the step of controlling the zoom factor of the field-of-view camera, to perform enlargement control processing in which the zoom factor of the field-of-view camera is increased. A position corresponding to an average value or a mode value of coordinate values representing positions of the viewpoint on the surface of the display unit obtained through the step of detecting the line of sight of the user is referred to as a neutral viewpoint position,
calculating the neutral viewpoint position;
When displaying the image acquired by the field-of-view camera on the display unit, displaying the image by aligning the center of the image acquired by the field-of-view camera with the neutral viewpoint position;
In the step of displaying the gaze icon superimposed on the image acquired by the field-of-view camera on the display unit, the process of displaying the gaze icon after providing the center area around the neutral viewpoint position is performed by the calculation. 8. The program according to claim 7, which is executed by a device.

Images