JP2024170159A - Electronic device and method for controlling electronic device - Google Patents

Abstract

To provide an electronic apparatus which properly transits a display state and a non-display state of a visual line position and thereby improves the visibility of the visual line position and a captured image.SOLUTION: The electronic apparatus comprises: acquisition means which acquires information on the visual line position of a user viewing a captured image; and control means which performs control to display an item at a position based on the visual line position of the user. The control means transits one of a display state and a non-display state of the visual line position to the other, by a transition system selected out of a plurality of transition systems.SELECTED DRAWING: Figure 8D

Description

The present invention relates to an electronic device and a method for controlling an electronic device.

Conventionally, imaging devices have been provided that are equipped with a function for detecting the gaze position of a user looking into the viewfinder field of view. In recent years, imaging devices have been provided that have electronic viewfinders that can display images captured by an image sensor that receives a light beam that has passed through a shooting optical system. Imaging devices with electronic viewfinders are capable of recognizing and detecting various types of subjects. Such imaging devices can perform focus detection control at a subject position that reflects the user's intentions, based on information about the detected gaze position and the results of subject recognition in the captured image.

Patent Document 1 discloses a technique for optimally controlling the display state of the gaze position. Patent Document 2 discloses a technique for displaying the gaze position during focus detection and photometry processing and the gaze position immediately before shooting in different display formats.

JP 2021-131441 A Japanese Unexamined Patent Publication No. 8-184746

However, if the display state and display format of the user's gaze position are changed without taking into account the imaging conditions and the conditions of the captured subject, the gaze position and visibility of the captured image may be reduced.

The present invention aims to provide an electronic device that improves the visibility of the gaze position and captured images by appropriately transitioning between a display state and a non-display state of the gaze position.

A first aspect of the present invention is an electronic device having an acquisition means for acquiring information on the gaze position of a user viewing a captured image, and a control means for controlling to display an item at a position based on the gaze position of the user, the control means transitioning the item from one of a display state and a non-display state to the other in a transition method selected from a plurality of transition methods.

A second aspect of the present invention is a method for controlling an electronic device, comprising an acquisition step for acquiring information on the gaze position of a user viewing a captured image, and a control step for controlling to display an item at a position based on the gaze position of the user, the control step being characterized in that the item is transitioned from one of a display state and a non-display state to the other in a transition method selected from a plurality of transition methods.

A third aspect of the present invention is a program for causing a computer to function as each of the means of the electronic device described above.A fourth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device described above.

The present invention provides an electronic device that improves the visibility of the gaze position and captured images by appropriately transitioning between a display state and a non-display state of the gaze position.

FIG. 1 is a diagram illustrating an example of the appearance of a camera. FIG. 2 is a diagram illustrating an example of the internal configuration of a camera. FIG. 2 is a block diagram showing an example of an electrical configuration within the camera. FIG. 4 is a diagram showing an example of a display in a viewfinder field of view. 1 is a diagram for explaining the principle of a gaze detection method. FIG. 2 shows an eye image. 13 is a flowchart of a gaze detection process. 11 is a flowchart illustrating an icon display control process. 13 is a flowchart of an icon display determination process based on an imaging situation. 13 is a flowchart of an icon display determination process based on a subject situation. 11 is a flowchart of an icon display process according to the first embodiment. 10 is a flowchart of an icon non-display process according to the first embodiment. FIG. 4 is a diagram illustrating subject movement information. FIG. 13 is a diagram illustrating an icon indicating a gaze position. FIG. 11 is a diagram illustrating icon display based on a zoom operation. 11A and 11B are diagrams illustrating icon display based on changes in camera attitude. 11A and 11B are diagrams illustrating icon display based on a change in the number of subjects. 11A and 11B are diagrams illustrating icon display based on subject movement information. 13A and 13B are diagrams illustrating an example in which an icon is displayed by changing the transparency. FIG. 13 is a diagram showing an example of changing the size of an icon to a displayed state. 10 is a flowchart of an icon display process according to the second embodiment. 10 is a flowchart of an icon non-display process according to the second embodiment. FIG. 13 is a diagram illustrating icon display for multiple subjects that are close to each other. FIG. 13 is a diagram illustrating icon display for multiple subjects that are not in close proximity to each other. 13 is a flowchart of an icon display process according to a third embodiment. 13 is a flowchart of an icon non-display process according to the third embodiment. FIG. 13 is an external view of another electronic device.

[Embodiment 1]
Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

<Description of configuration>
An example of the camera configuration will be described with reference to Figs. 1(a), 1(b) to 3. Figs. 1(a) and 1(b) show an example of the appearance of a camera 1 (digital still camera; interchangeable lens camera) according to an embodiment. Fig. 1(a) is a front perspective view, and Fig. 1(b) is a rear perspective view. As shown in Fig. 1(a), the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operating member capable of accepting photographing operations from a user (photographer), is arranged on the camera housing 1B.

As shown in FIG. 1B, an eyepiece lens 12 (eyepiece optical system) into which a user looks to view a display device 10 (display panel) included in the camera housing 1B is disposed on the rear surface of the camera housing 1B. The eyepiece optical system may include a plurality of lenses. Operation members 41 to 43 capable of accepting various operations from a user are also disposed on the rear surface of the camera housing 1B. For example, the operation member 41 is a touch panel capable of accepting touch operations, the operation member 42 is an operation lever that can be pushed down in each direction, and the operation member 43 is a four-way key that can be pushed in each of four directions. The operation member 41 (touch panel) is provided with a display panel such as a liquid crystal panel, and has a function of displaying an image on the display panel.

Figure 2 is a cross-sectional view of camera 1 cut along the YZ plane defined by the Y axis and Z axis shown in Figure 1(a), and shows an example of the internal configuration of camera 1. In Figure 2, the same components as in Figures 1(a) and 1(b) are denoted by the same reference numerals.

The photographing lens unit 1A includes two lenses 101, 102, an aperture 111, an aperture drive section 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. The photographing lens unit 1A also includes an acceleration sensor 119, a zoom drive section 120, and a lens CPU 121. Note that for simplicity, two lenses 101, 102 are shown, but the photographing lens unit 1A actually includes more than two lenses. Each component included in the photographing lens unit 1A is controlled by the lens CPU 121.

Aperture 111 provided in photographing lens unit 1A is driven by aperture drive unit 112. Lens drive member 114 includes a drive gear and the like. Photocoupler 115 detects the rotation of pulse plate 116 linked to lens drive member 114 and transmits it to focus adjustment circuit 118. Focus adjustment circuit 118 drives lens drive motor 113 based on information from photocoupler 115 and information from camera housing 1B (lens drive amount information), and moves lens 101 to change the focus position. Mount contact 117 is an interface between photographing lens unit 1A and camera housing 1B.

The acceleration sensor 119 is built into the photographing lens unit 1A and detects the panning of the lenses 101 and 102. The acceleration sensor 6 is built into the camera housing 1B and detects the panning of the camera housing 1B. The panning of the camera 1 is determined using at least one of the acceleration sensor 119 built into the photographing lens unit 1A and the acceleration sensor 6 built into the camera housing 1B.

The zoom drive unit 120 performs zoom drive based on manual operation by the user using an operating member provided on the lens barrel (not shown), or performs zoom drive based on instructions from the camera housing 1B notified via the mount contact 117. The lens CPU 121 notifies the camera housing 1B of zoom drive information from the photographing lens unit 1A via the mount contact 117. The camera 1 can detect a change in the angle of view due to zoom operation based on the zoom drive information.

The camera housing 1B includes an image sensor 2, a CPU 3, a memory section 4, a display device 10, a display device drive circuit 11, etc. The image sensor 2 is disposed at the intended imaging surface of the photographing lens unit 1A. The CPU 3 is the central processing section of the microcomputer, and controls the entire camera 1. The memory section 4 stores images captured by the image sensor 2, etc. The display device 10 is composed of a liquid crystal or the like, and displays the captured image (subject image) etc. on the display surface of the display device 10. The display device drive circuit 11 drives the display device 10. The user can view the image (such as the image captured by the image sensor 2) displayed on the display surface of the display device 10 through the eyepiece 12.

The camera housing 1B also includes an eyepiece 12, light sources 13a and 13b, a light splitter 15, a light receiving lens 16, and an ocular imaging element 17. The light sources 13a and 13b are light sources for illuminating the user's eyeball 14. The light sources 13a and 13b are used in single-lens reflex cameras and the like to detect the line of sight (the direction of the line of sight; the direction in which the user is looking) from the relationship between the reflection image (corneal reflection image; Purkinje image) of light due to the corneal reflection and the pupil. Specifically, the light sources 13a and 13b are infrared light-emitting diodes or the like that emit infrared light that is insensitive to the user, and are arranged around the eyepiece 12.

The optical image of the illuminated eyeball 14 (eyeball image; an image formed by light emitted from light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the light splitter 15. The eyeball image is then formed by the light receiving lens 16 on the ocular imaging element 17, which is a two-dimensional array of photoelectric elements such as CCD or CMOS. The light receiving lens 16 positions the pupil of the eyeball 14 and the ocular imaging element 17 in a conjugate imaging relationship. The line of sight of the eyeball 14 is detected by a specified algorithm from the positional relationship between the pupil and the corneal reflection image in the eyeball image formed on the ocular imaging element 17.

Figure 3 is a block diagram showing an example of the electrical configuration within the camera 1. In Figure 3, the same components as those in Figures 1(a), 1(b) and 2 are given the same reference numerals, and detailed descriptions of these components are omitted. The CPU 3 is connected to the line of sight detection circuit 201, the photometry circuit 202, the autofocus detection circuit 203, the signal input circuit 204, the display device drive circuit 11, the light source drive circuit 205, the acceleration sensor 6, the memory unit 4, the image processing unit 208 and the attitude detection unit 207.

The CPU 3 also transmits a signal to the lens CPU 121 via the mount contact 117. Based on the signal from the CPU 3, the lens CPU 121 controls the focus adjustment circuit 118 arranged in the photographing lens unit 1A, the aperture control circuit 206 included in the aperture drive unit 112, the acceleration sensor 119, and the zoom drive unit 120.

The memory unit 4 associated with the CPU 3 has a function of storing image signals from the image sensor 2 and the eye image sensor 17. The CPU 3 converts the image signal of the image sensor 2 stored in the memory unit 4 into a captured image, and transfers the converted captured image to the recording medium 7 connected to the memory unit 4.

The attitude detection unit 207 detects attitude information of the user holding the camera 1. The attitude detection unit 207 can detect the attitude information of the user using, for example, the acceleration sensor 6. The camera 1 can capture images in a normal position (horizontal position) where the image sensor 2 is horizontal to the subject, and in a vertical position where the image sensor 2 is vertical to the subject. The attitude detection unit 207 can detect a change in the attitude of the camera 1 even if the user changes the attitude of the camera 1 while looking through the viewfinder.

The gaze detection circuit 201 A/D converts the output of the eye imaging element 17 (eye image captured of the eye) when an eyeball image is formed on the eye imaging element 17, and transmits the result to the CPU 3. The CPU 3 extracts feature points used for gaze detection from the eye image according to a predetermined algorithm, and calculates the user's viewpoint (gaze position; position where the gaze is directed; position where the user is looking) on the display surface of the display device 10 from the positions of the feature points.

The photometry circuit 202 amplifies, logarithmically compresses, and A/D converts the signal obtained from the image sensor 2, which also functions as a photometry sensor; specifically, the luminance signal corresponding to the brightness of the field, and sends the resulting field luminance information to the CPU 3. The field luminance information is information indicating the luminance of each pixel in the captured image. The photometry circuit 202 also detects a specific subject (specific object) from the image captured by the image sensor 2, and sends the resulting subject information and subject area information to the CPU 3. The subject information (object information) is information related to the detected subject,
The subject area information (object area information) is information that indicates the area of the detected subject. The subject information and the subject area information can also be obtained by analyzing the field luminance information.

The auto focus detection circuit 203 A/D converts the signal voltage from multiple detection elements (multiple pixels) used for phase difference detection, which are included in the image sensor 2, and sends it to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals of the multiple detection elements. This is a well-known technique known as image plane phase difference AF. In this embodiment, as an example, it is assumed that there are focus detection points in the viewfinder field (the field of view when looking through the viewfinder) as shown in FIG. 4(a), specifically, at each of the 180 locations on the image plane corresponding to the 180 locations shown on the display surface of the display device 10.

Switches SW1 and SW2 are connected to the signal input circuit 204. Switch SW1 is turned on by the first stroke (half press) of the release button 5, and is a switch for starting the exposure metering and focus detection (distance measurement) of the camera 1. Switch SW2 is turned on by the second stroke (full press) of the release button 5, and is a switch for starting the shooting operation. ON signals from switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3. Exposure metering and focus detection are performed within a set exposure metering area or focus detection area, or within the area of a selected (determined) main subject. For example, in exposure metering, the exposure metering circuit 202 determines the brightness of the target area from the field brightness information, and in focus detection, the auto focus detection circuit 203 determines the distance to the subject present in the target area.

Operation member 41 (touch panel), operation member 42 (operation lever), and operation member 43 (four-way key) send operation signals corresponding to operations received from the user to CPU 3. Light source drive circuit 205 drives light sources 13a and 13b.

The image processing unit 208 applies various image processing to the image captured by the image sensor 2. The image processing unit 208 also acquires and generates various information from the captured image. The image processing unit 208 may be realized by a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. The image processing unit 208 may also be realized by a processor such as a DSP (Digital Signal Processor) that realizes a specific function by executing software.

Image processing by the image processing unit 208 includes pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, etc. Pre-processing includes processing such as signal amplification, reference level adjustment, and defective pixel correction. Color interpolation processing is processing that interpolates values of color components not included in the image data, and is also called demosaic processing. Correction processing includes processing that adjusts white balance, processing that corrects image brightness, processing that corrects optical aberrations of the photographing lens unit 1A, and processing that corrects color.

The detection process includes detection of characteristic regions (e.g., regions of faces, human bodies, animals, automobiles, etc.), tracking of characteristic regions, person recognition, etc. The image processing unit 208 detects information on the user's gaze position on the display surface of the display device 10 that displays the image, and detects the subject within the image, thereby being able to identify the subject that the user is focusing on. In addition, the image processing unit 208 can obtain information on the movement characteristics of the subject, such as how the subject in the image is moving on the image plane, by analyzing the difference between images over time.

The data processing includes scaling, encoding and decoding, header information generation, etc. The evaluation value calculation includes calculation of a pair of signals for AF processing using a phase difference signal, an evaluation value for contrast AF, an evaluation value used for automatic exposure control, etc. Note that the image processing performed by the image processing unit 208 is not limited to the above examples.

Figures 4(a) and 4(b) are diagrams showing examples of the display of the field of view in the viewfinder, showing the state in which the display device 10 is operating (the state in which an image is displayed). As shown in Figure 4(a), the field of view in the viewfinder includes a field of view mask 300, a focus detection area 400, and 180 ranging point indices 401. Each of the 180 ranging point indices 401 is displayed superimposed on a through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to the focus detection point on the imaging surface. The image processing unit 208 can obtain information on the user's gaze position on the display surface of the display device 10 from the eye image obtained by the ocular imaging element 17 when the user looks through the viewfinder.

In FIG. 4(a), the gaze position 421 is the position where the user's gaze was detected (the area including the gaze position). The subject 422 is a person detected by the image analysis by the image processing unit 208. In the following description, when the detection target is a person, the hatched head area is detected as the subject. Since the human eyeball undergoes a movement accompanied by a fluctuation called fixation eye movement, it is preferable for the image processing unit 208 to estimate the subject that exists near the gaze position as the subject at which the user's gaze is directed. When there is a discrepancy between the gaze position and the position at which the subject exists, the image processing unit 208 displays an icon 423 indicating the subject at which the user's gaze is directed. By displaying the icon 423, the user can confirm the subject that is estimated to be viewed by the user. Furthermore, the image processing unit 208 can estimate the subject that the user is viewing by directing the user's gaze near the desired subject.

The vicinity of the gaze position can be a predetermined range including the gaze position. The predetermined range is determined, for example, based on the focal length, shooting distance, shooting mode, and the size of the subject at which the gaze is directed (present at a position based on the user's gaze position) when capturing the image. The size of the subject is the size within the image, and can be expressed, for example, by the number of pixels the subject occupies. The predetermined range may also be determined based on the type of subject, the movement speed, etc. When multiple subjects are detected near the gaze position, the image processing unit 208 can estimate that the subject closest to the gaze position is the subject at which the user's gaze is directed.

Figure 4(b) shows an example in which multiple subjects are present in an image. In the image shown in Figure 4(b), subjects 431 and 432 are detected. Gaze position 433 is the position where the user's gaze is detected. Icon 434 (corresponding to the first item) indicates the subject (hereinafter also referred to as the estimated subject) that is estimated by image processing unit 208 to be directed at by the user's gaze.

When the user moves the line of sight to the vicinity of subject 432, image processing unit 208 changes the estimated subject from subject 431 to subject 432. Image processing unit 208 displays icon 434 indicating the estimated subject for subject 432, the destination of the line of sight. Note that the subject detected in the image is not limited to a person, and may be an animal, a vehicle, or the like.

<Description of Gaze Detection Processing>
The gaze detection process (gaze detection method) for acquiring the gaze position will be described with reference to Figs. 5, 6(a), 6(b), and 7. The gaze of the right eye and the gaze of the left eye are detected by the following gaze detection method. Fig. 5 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for detecting the gaze. As shown in Fig. 5, the light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of the light receiving lens 16, and illuminate the user's eyeball 14. A part of the light emitted from the light sources 13a and 13b and reflected by the eyeball 14 is collected by the light receiving lens 16 on the eye image sensor 17. Fig. 6(a) is a schematic diagram of an eye image captured by the eye image sensor 17 (optical image of the eye projected on the eye image sensor 17), and Fig. 6(b) is a diagram showing the output intensity of the eye image sensor 17. Fig. 7 is a flowchart of the gaze detection process.

When the gaze detection process in FIG. 7 starts, in step S1, the CPU 3 drives the light sources 13a and 13b using the light source drive circuit 205 so as to emit infrared light toward the user's eyeball 14. An optical image of the user's eye illuminated by the infrared light passes through the light receiving lens 16 and is formed on the eye image sensor 17, which then performs photoelectric conversion. This provides an electrical signal of the eye image that can be processed.

In step S2, the CPU 3 acquires an eye image (image data, image signal) from the eye imaging element 17 via the gaze detection circuit 201.

In step S3, the CPU 3 obtains the corneal reflection images Pd, Pe of the light sources 13a, 13b and the coordinates of the point corresponding to the center c of the pupil 141 from the eye image obtained in S2. The infrared light emitted from the light sources 13a, 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the corneal reflection images Pd, Pe formed by a portion of the infrared light reflected from the surface of the cornea 142 are collected by the light receiving lens 16 and imaged on the ocular imaging element 17 to become the corneal reflection images Pd', Pe' in the eye image. Similarly, the light beams from the ends a, b of the pupil 141 are also imaged on the ocular imaging element 17 to become the pupil end images a', b' in the eye image.

Figure 6(b) shows the luminance information (luminance distribution) of region α in the eye image of Figure 6(a). In Figure 6(b), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. The coordinates in the X-axis direction (horizontal direction) of the corneal reflection images Pd', Pe' are Xd, Xe, and the coordinates in the X-axis direction of the pupil edge images a', b' are Xa, Xb.

As shown in FIG. 6B, the coordinates Xd and Xe of the corneal reflection images Pd' and Pe' provide an extremely high level of luminance. In the region from coordinate Xa to coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by focusing the light beam from the pupil 141 on the ocular imaging element 17), an extremely low level of luminance is provided except for coordinates Xd and Xe. In the region of the iris 143 outside the pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from the iris 143), an intermediate luminance between the above two types of luminance is provided. For example, an intermediate luminance between the above two types of luminance is provided in a region where the X coordinate (coordinate in the X-axis direction) is greater than coordinate Xa and a region where the X coordinate is less than coordinate Xb.

From the luminance distribution as shown in FIG. 6(b), the X coordinates Xd, Xe of the corneal reflection images Pd', Pe' and the X coordinates Xa, Xb of the pupil edge images a', b' can be obtained. For example, the coordinates of the corneal reflection images Pd', Pe' can be obtained as the coordinates of the corneal reflection images Pd', Pe', and the coordinates of the pupil edge images a', b' can be obtained as the coordinates of the pupil edge images a', b'. In addition, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by focusing the light beam from the pupil center c on the ocular imaging element 17 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this way, the coordinates of the corneal reflection images Pd', Pe' and the coordinates of the pupil center image c' can be estimated.

In step S4, the CPU 3 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of the eyeball 14 relative to the light receiving lens 16, and can be calculated using a function of the distance (Xd-Xe) between the corneal reflection images Pd', Pe'.

In step S5, CPU 3 calculates the rotation angle of the optical axis of eyeball 140 relative to the optical axis of light receiving lens 16. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 142 approximately coincide with each other. For this reason, if the standard distance from the center of curvature O of cornea 142 to the center of pupil c is Oc, then the rotation angle _θX of eyeball 14 in the Z-X plane (plane perpendicular to the Y axis) can be calculated by the following formula 1. The rotation angle θy of eyeball 140 in the Z-Y plane (plane perpendicular to the X axis) can also be calculated by a method similar to the method for calculating the rotation angle θx.

β*Oc*SINθ _X ≒ {(Xd+Xe)/2}-Xc... (Formula 1)

In step S6, the CPU 3 uses the rotation angles θx, θy calculated in step S5 to estimate the user's line of sight (hereinafter also referred to as the viewpoint) on the display device 10. If the coordinates (Hx, Hy) of the viewpoint correspond to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following formulas 2 and 3.

Hx=m×(Ax×θx+Bx) (Formula 2)
Hy=m×(Ay×θy+By) (Formula 3)

The parameter m in Equations 2 and 3 is a constant determined by the configuration of the optical system for performing gaze detection processing, and is a conversion coefficient that converts the rotation angles θx and θy into coordinates corresponding to the pupil center c on the display device 10. The parameter m is determined in advance and stored in the memory unit 4. The gaze correction parameters Ax, Bx, Ay, and By are gaze correction parameters that correct individual differences in the gaze. The gaze correction parameters Ax, Bx, Ay, and By are obtained by performing calibration of the gaze detection, and are stored in advance in the memory unit 4.

In step S7, the CPU 3 stores the coordinates of the viewpoint and the time when the eye image was acquired (hereinafter, the gaze detection time) in the memory unit 4, and ends the gaze detection process.

The gaze detection method is not limited to the above method, and any method that detects the gaze based on an eye image may be used. Also, an example has been described in which the coordinates of the viewpoint (Hx, Hy) are obtained as the result (final result) of gaze detection, but any information related to the gaze (gaze information), such as rotation angles θx, θy, may be obtained as the result of gaze detection.

<Description of Icon Display Control Process>
8A to 8C, the display control of an icon indicating a subject at which the user's gaze is directed will be described. Fig. 8A is a flowchart illustrating an example of an icon display control process in the first embodiment. The process shown in Fig. 8A includes image acquisition, subject detection, and icon display control processes. The CPU 3 acquires images in chronological order, and repeatedly executes the process shown in Fig. 8A.

In step S8001, the CPU 3 acquires an image captured by the image sensor 2 and generated by the image processing unit 208 through various image processing. In step S8002, the CPU 3 detects a subject included in the image acquired in step S8001. Specifically, the CPU 3 acquires information on the position of the head of a person detected from the image as subject position information. For example, in the image of FIG. 9(a), the CPU 3 detects the subject 901 and acquires information on the head position of the subject 901.

In step S8003, the CPU 3 acquires motion information (motion characteristic information, action information) of the subject included in the image. The CPU 3 can acquire motion information of the subject included in the image by comparing the image acquired in step S8001 with an image previously generated by the image processing unit 208. For example, FIGS. 9(a) to 9(c) are images generated at time t1, time t2 after time t1, and time t3 after time t2, respectively. In this case, the CPU 3 can acquire motion information that the subject 901 is moving at a substantially constant speed. Also, as in FIG. 9(d), when multiple subjects including the subject 901 are detected in the image and all of them show similar motion characteristics, these multiple subjects are treated as a group of subjects 902.

In step S8004, CPU 3 performs the gaze detection process described in Figures 5 to 7 using the eye imaging element 17 and gaze detection circuit 201 to acquire gaze position information. In step S8005, CPU 3 determines whether gaze detection (acquisition of gaze position information) in step S8004 was successful. If gaze detection was successful, processing proceeds to step S8007. If gaze detection was unsuccessful, processing proceeds to step S8006.

In step S8006, CPU 3 maintains the display position of the icon (first item) in the previous process (processing for the image captured before the current image) and the display state of whether to display it, and returns to step S8001. Specifically, if the icon was not displayed in the previous process, CPU 3 controls so that the icon is not displayed for the current image either, and if the icon was displayed in the previous process, the icon is displayed for the current image in the same position as in the previous image.

If gaze detection fails, the CPU 3 may display the icon in a different manner than when gaze detection is successful, in order to notify the user that gaze detection has failed. For example, if gaze detection fails, the CPU 3 may display the icon in a different color than when gaze detection is successful, or may blink the icon. Note that, in order to prevent visibility from decreasing due to frequent changes in the icon color, the CPU 3 may change the display manner of the icon after a predetermined time has elapsed since it was determined that gaze detection has failed.

In step S8007, CPU 3 determines whether or not a subject has been detected at the gaze position acquired in step S8004. If a subject detected in step S8002 is present near the user's gaze position, CPU 3 can determine that a subject has been detected at the gaze position. If a subject has been detected at the gaze position, processing proceeds to step S8009. If a subject has not been detected at the gaze position, processing proceeds to step S8008.

In step S8008, CPU 3 displays an icon at the gaze position acquired in step S8004. For example, in FIG. 10, the user's gaze position is located away from subject 1001 included in the image, so CPU 3 displays icon 1002 (second item) indicating the detected user's gaze position. For example, when subject 1001 included in the image is away from the user's gaze position by a predetermined distance or more, CPU 3 displays an icon indicating the user's gaze position.

When displaying icon 1002 indicating the gaze position, because there is no subject at the user's gaze position, CPU 3 does not display an icon (first item) indicating the subject at which the user's gaze is directed. This allows CPU 3 to inform the user that subject 1001 has not been captured as the subject at which the user's gaze is directed.

Area 1003 is an area indicating the vicinity of the user's gaze position. CPU 3 sets area 1003 as an area for determining whether or not a subject is present in the vicinity of the gaze position. Area 1003 may be set by the user using operation members 41 to 43. CPU 3 may also automatically set area 1003 according to the optical state of the lens, such as the focal length of photographing lens unit 1A or the photographing object distance at the in-focus position, or may automatically set area 1003 according to the mode setting of the AF function set in camera 1. In other words, the vicinity of the gaze position is an area that includes the user's gaze position and can be set by the user or CPU 3.

In step S8009, CPU 3 determines whether or not to display an icon indicating the subject at which the user's gaze is directed, based on the status of the user's image capture with camera 1. The status of the image capture with camera 1 includes, for example, statuses such as whether or not the user is paying attention to the subject estimated by image processing unit 208, whether or not the image capture range of the image has changed, and whether or not the attitude of the image capture device capturing the image has changed.

The process of step S8009 will be described in detail with reference to FIG. 8B. In step S8201, CPU 3 determines whether the user, who is the photographer, is paying attention to the subject that he or she wishes to photograph (the subject that is estimated to be directed at, the estimated subject).

Whether or not the user is looking at the estimated subject is determined from the eye image captured by the eye image sensor 17 based on the degree of stability of the user's gaze position, the presence or absence or frequency of the user's blinking, etc. For example, the CPU 3 can determine that the user is looking at the estimated subject when the amount of movement of the user's gaze position in a specified time is smaller than a first threshold value, or when the number of times the user blinks in a specified time is smaller than a second threshold value. Since the degree of stability of gaze position and blinking frequency vary from person to person, the first threshold value and the second threshold value may be set in advance for each user. If the user is looking at the estimated subject, processing proceeds to step S8202. If the user is not looking at the estimated subject, processing proceeds to step S8203.

In step S8202, CPU 3 determines that an icon indicating the estimated subject is to be displayed. By displaying an icon indicating the estimated subject, CPU 3 can notify the user that the subject the user is focusing on has been captured as an image capture target. For example, when the user is waiting for a subject that the user wishes to photograph at a dolphin show to move to a desired position, or when the user is waiting for a photo opportunity in front of a soccer goal, the user can avoid missing a photo opportunity by displaying an icon indicating the subject the user is focusing on.

In step S8203, CPU 3 determines whether or not a zoom operation, pan operation, or tilt operation has occurred. Zoom operations include zoom operations performed by the user on the photographing lens unit 1A and digital zoom operations performed on camera 1. CPU 3 determines whether or not a zoom operation, pan operation, or tilt operation has occurred, thereby determining whether or not the imaging range of the image has changed.

The CPU 3 can determine that the imaging range of the image has changed, for example, when a zoom operation has been performed, when the amount of change in focal length due to the zoom operation is greater than a third threshold, or when the amount of change in zoom magnification due to the zoom operation is greater than a fourth threshold. The CPU 3 can also determine that the imaging range of the image has changed, when a panning operation or tilting operation has been performed, or when the amount of movement of the imaging range due to the panning operation or tilting operation is greater than a fifth threshold. The third to fifth thresholds can be determined, for example, based on whether or not there is a possibility that the subject on which the user is directing his or her gaze will be lost due to a change in the imaging range. If a zoom operation, panning operation, or tilt operation has been performed, the process proceeds to step S8202. If a zoom operation, panning operation, or tilt operation has not been performed, the process proceeds to step S8204.

In step S8202, CPU 3 determines to "display an icon" indicating the estimated subject. When the composition is changed by a zoom operation or the like, the position of the subject that the user was directing his or her gaze may change. If a different subject comes to exist in the position of the subject that the user was originally directing his or her gaze, the user may lose sight of the subject that the user was trying to photograph. Therefore, when the imaging range of the image has changed by a zoom operation or the like, CPU 3 displays an icon indicating the estimated subject, thereby preventing the user from losing sight of the subject that the user was trying to photograph.

Figures 11(a) and 11(b) are diagrams explaining icon display based on a zoom operation. Figure 11(a) shows a state in which, when zoomed in to the TELE side, subject 1101 is estimated as the subject at which the user's gaze is directed. Since there is only one subject near gaze position 1102 (the icon indicating gaze position 1102 is not displayed on the display device 10), the icon indicating the subject at which the user's gaze is directed is not displayed.

Figure 11 (b) shows the situation after zooming in the WIDE direction from the state of Figure 11 (a). In Figure 11 (b), a subject 1103 different from subject 1101 is detected in the image. If gaze position 1102 is the same as before the zoom operation, the user may be confused as to whether subject 1103 has been captured as an estimated subject. Therefore, when the user is directing their gaze at subject 1101, CPU 3 determines to display an icon for subject 1101 to inform the user that subject 1101 has been captured as an estimated subject.

In step S8204, CPU 3 determines whether the attitude of camera 1 has changed. When taking pictures with camera 1, the user alternates between a horizontal position in which the image is taken with the long side oriented horizontally, and a vertical position in which the image is taken with the long side oriented vertically. CPU 3 determines whether the attitude of camera 1 has changed, for example, by changing the camera from a horizontal position to a vertical position. CPU 3 can obtain information on the attitude of camera 1 using attitude detection unit 207.

The CPU 3 may determine whether the attitude of the camera 1 has changed after acquiring information on the user's gaze position. If the attitude of the camera 1 is changed while looking through the viewfinder, an icon indicating the estimated subject is displayed, making it easier for the user to confirm the desired subject. If the attitude of the camera 1 has changed, the process proceeds to step S8202. If the attitude of the camera 1 has not changed, the process proceeds to step S8205.

In step S8202, CPU 3 determines to display an icon indicating the estimated subject. This is because when the attitude of camera 1 changes, the user's line of sight is likely to become unstable, and this is done to prevent the user from losing sight of the subject that was being viewed before the attitude of camera 1 changed.

Figures 12(a) and 12(b) are diagrams explaining icon display based on a change in the posture of camera 1. Figure 12(a) shows an example of an image captured in a horizontal position. In Figure 12(a), subjects 1201 and 1202 are detected. The user's gaze is directed at subject 1201. Figure 12(b) shows an example of an image after the user changes the position of camera 1 from the state shown in Figure 12(a) to a vertical position. While camera 1 is changed from the horizontal position to the vertical position, the user's gaze position becomes unstable due to the change in composition. To prevent the user from mistakingly recognizing subject 1202 as subject 1201 that was previously viewed, CPU 3 determines to display icon 1203 indicating the estimated subject.

In step S8205, CPU 3 determines that it is undetermined whether or not to display an icon indicating the estimated subject ("icon display is undetermined"). In other words, if none of the conditions in steps S8201, S8203, and S8204 is satisfied, CPU 3 determines whether or not to display an icon indicating the estimated subject based on the state of the subject in step S8020 in Fig. 8A described later.

In step S8206, CPU 3 determines whether the icon indicating the estimated subject is currently in a displayed or hidden state. If the icon indicating the estimated subject is in a hidden state, processing proceeds to step S8207. If the icon indicating the estimated subject is in a displayed state, the processing shown in FIG. 8B ends.

In step S8207, CPU 3 records the current time as a reference time in memory unit 4. Step S8207 is a process performed in the case where the icon representing the estimated subject is currently in a non-display state and it is determined in step S8202 that the icon is to be displayed. By recording the time (reference time) at which it is determined that the icon will transition from a non-display state to a display state, CPU 3 can measure the time since the icon transitioned to a display state.

As described above, by the process shown in FIG. 8B, CPU 3 determines whether or not to display an icon of a subject at which the user's gaze is directed, based on the imaging situation (the situation of the photographer). CPU 3 ends the process in FIG. 8B and proceeds to step S8010 in FIG. 8A.

In step S8010 of FIG. 8A, CPU 3 determines whether the result of the display determination of the icon indicating the estimated subject in step S8009 was "display icon." If the result of the determination is "display icon," processing proceeds to step S8011. If the result of the determination is "icon display undetermined," processing proceeds to step S8020.

In step S8011, CPU 3 determines whether or not a predetermined time has elapsed since the reference time recorded in step S8207 of FIG. 8B. That is, CPU 3 determines whether or not a predetermined time has elapsed since the time (reference time) when the icon representing the estimated subject transitioned from a non-display state to a display state. If the predetermined time has elapsed since the reference time, processing proceeds to step S8013. If the predetermined time has not elapsed since the reference time, processing proceeds to step S8012.

In step S8012, CPU 3 displays an icon indicating the estimated subject (e.g., icon 423 in FIG. 4(a)). In step S8013, CPU 3 hides the icon indicating the estimated subject. After a predetermined time has elapsed since the imaging conditions changed and the icon indicating the estimated subject was displayed, it is expected that the user will be able to stably direct their gaze toward the desired subject, so CPU 3 hides the icon indicating the estimated subject.

In the above example, the icon indicating the estimated subject is automatically hidden when a predetermined time has elapsed since it was switched from a hidden state to a visible state, but the CPU 3 may maintain the icon's visible state. Also, the predetermined time from the reference time until the icon indicating the estimated subject is hidden may be set by the user. In steps S8012 and S8013, when the icon indicating the estimated subject is respectively switched to a visible state and a hidden state, the process returns to step S8001.

In step 8020, CPU 3 determines whether to display an icon indicating the subject at which the user's gaze is directed, based on the subject's status. The subject's status includes, for example, whether the number of subjects included in the image has changed, whether the number of subjects included in the image is two or more, whether the estimated subject is accelerating or decelerating, and whether the estimated subject is in focus.

The details of the process of step S8020 will be described with reference to FIG. 8C. In step S8301, the CPU 3 determines whether or not the number of subjects in the image has increased or decreased since the previous image (frame). The number of subjects is not limited to the number of subjects in the image, and may be the number of subjects detected in the vicinity of the gaze position. Specifically, the number of subjects can be the number of subjects detected in a predetermined range including the estimated subject to which the gaze is directed. The predetermined range can be determined based on at least one of the focal length, shooting distance, shooting mode, and size of the estimated subject when capturing the image, for example. The subject to be detected is a person, an animal, a car, or the like that can be detected by analyzing the image.

The CPU 3 may detect a predetermined type of subject from among the subjects included in the image. For example, if the user sets animals as the predetermined type, the CPU 3 detects animals as the subject. The user may set multiple types of subjects as the predetermined type. If there is an increase or decrease in the number of detected subjects, the process proceeds to S8302. If there is no increase or decrease in the number of detected subjects, the process proceeds to S8303.

In step S8302, CPU 3 determines to display an icon indicating the estimated subject. By displaying an icon indicating the estimated subject, the user can easily confirm the subject that he or she is viewing, even if the number of subjects increases or decreases.

In step S8303, CPU 3 determines whether or not the number of subjects detected near the gaze position is one. If the number of subjects is one, processing proceeds to step S8304. If the number of subjects is multiple, processing proceeds to step S8307.

In step S8304, CPU 3 determines whether or not one subject existing in the vicinity of the gaze position on the screen is in focus. If the subject is in focus, processing proceeds to step S8305. If the subject is not in focus, processing proceeds to step S8306.

In step S8305, CPU 3 determines to "hide the icon." In step S8306, CPU 3 determines to "display the icon." By displaying the icon when the subject the user is looking at is out of focus, CPU 3 can notify the user that the subject is not in focus.

Figures 13(a) and 13(b) are diagrams explaining icon display based on the change in the number of subjects described in the processing of steps S8301 to S8306. Figure 13(a) shows a situation where, at time t11, subject 1301 is present near gaze position 1302. Because it is clear that the subject 1301 is the object toward which the user's gaze is directed, the icon indicating the estimated subject and the icon indicating gaze position 1302 are hidden.

Figure 13(b) shows a state in which, following the situation in Figure 13(a), a different subject 1303 has entered the composition. Because subject 1303 has appeared near gaze position 1304, it is preferable for CPU 3 to notify the user of whether the gaze was directed toward subject 1301 or subject 1303. Therefore, CPU 3 displays icon 1305 indicating the subject toward which the user's gaze was directed. Note that the icon indicating gaze position 1304 is not displayed.

In step S8307, CPU 3 determines whether or not a subject present near the user's gaze position is stationary or moving at a constant speed. Step S8307 is a process that is executed when multiple subjects are present near the gaze position. CPU 3 acquires information on the movement characteristics of the subjects near the gaze position (movement information). The movement information can be acquired in the same manner as in step S8003 of FIG. 8A, which was described using FIGS. 9(a) to 9(c). Depending on the movement characteristics (movement) of the subject, it is preferable to display an icon indicating the subject at which the user's gaze is directed, so that the user does not lose sight of the desired subject.

If CPU 3 determines based on the motion information that the subject in the vicinity of the gaze position is stationary or moving at a constant speed, CPU 3 proceeds to step S8308. On the other hand, if CPU 3 determines that any of the subjects in the vicinity of the gaze position is moving while accelerating or decelerating, CPU 3 proceeds to step S8310.

In step S8308, CPU 3 determines whether the subject the user is looking at is in focus. If the subject the user is looking at is in focus, processing proceeds to step S8309. If the subject the user is looking at is out of focus, processing proceeds to step S8310.

In step S8309, CPU 3 determines to "hide the icon." If the subject is in focus, it is assumed that the user's line of sight is steadily directed toward the desired subject, and therefore it is determined that the icon indicating the estimated subject is to be hidden.

In step S8310, CPU 3 determines to "display an icon." If the subject is out of focus, it is assumed that the user is not stable, and so it is determined to display an icon indicating the presumed subject. Also, if a subject present in the vicinity of the gaze position is moving while accelerating or decelerating (for example, when performing vigorous movements such as dancing), CPU 3 determines to display an icon indicating the subject at which the user is directing his or her gaze. By displaying an icon indicating the subject at which the user is directing his or her gaze, the user is less likely to lose sight of the desired subject.

Figures 14(a) to 14(d) are diagrams explaining icon display based on subject movement information described in the processing of steps S8307 to S8310. Figure 14(a) is an example of so-called portrait photography, in which multiple subjects are stationary. Subject 1401 detected from the image is assumed to be out of focus. CPU 3 estimates that the user's gaze is directed at subject 1401 that exists near gaze position 1402, and displays icon 1403 indicating that subject 1401 is the estimated subject.

Figure 14 (b) shows a situation where subject 1401 is in focus. CPU 3 displays icon 1404 indicating that subject 1401 is in focus, and does not display icon 1403 indicating that it is an estimated subject. By giving priority to displaying icon 1404 indicating that the subject is in focus, CPU 3 can reduce the amount of information to be displayed and ensure visibility. CPU 3 can also notify the user in a timely manner that subject 1401 is in focus.

Figure 14(c) shows a situation where a subject group 1414 including a subject 1411 near the gaze position and multiple surrounding subjects have similar movement characteristics. Figure 14(c) shows the situation at time t21. CPU 3 displays an icon 1413 indicating that subject 1411 near gaze position 1412 is an estimated subject.

Figure 14(d) shows a situation after the situation in Figure 14(c) where a group of subjects 1414 including subject 1411 and multiple surrounding subjects are moving in the same direction at a substantially constant speed. Figure 14(d) shows the situation at time t22. At this time, a subject 1415 different from subject 1411 is present near gaze position 1412. Therefore, the subject at which the user's gaze is directed is estimated to be subject 1415.

In FIG. 14D, the subject 1415 is in focus. At time t22, the subject 1411 and the subject 1415 included in the subject group 1414 move with similar movement characteristics. The CPU 3 displays an icon 1416 indicating that the subject 1415 is in focus, but does not display an icon indicating that the subject is an estimated subject. The CPU 3 does not display an icon indicating that the subject 1415 is an estimated subject, and instead displays the icon 1416 preferentially, thereby suppressing the amount of information to be displayed and ensuring visibility. The CPU 3 can also notify the user in a timely manner that the subject 1415 has become in focus.

In step S8312, CPU 3 determines whether the process in steps S8301 to S8310 has determined that the icon should be displayed or that the icon should be hidden. If it has been determined that the icon should be displayed, the process proceeds to step S8313. If it has been determined that the icon should be hidden, the process shown in FIG. 8C ends.

In step S8313, CPU 3 determines whether the icon indicating the estimated subject is currently in a displayed state or a hidden state. If the icon indicating the estimated subject is in a hidden state, processing proceeds to step S8314. If the icon indicating the estimated subject is in a displayed state, the processing shown in FIG. 8C ends.

In step S8314, CPU 3 records the current time as a reference time in memory unit 4. Step S8314 is a process performed when the icon representing the estimated subject is currently in a non-display state and it is determined that the icon should be "displayed." By recording the time (reference time) at which it is determined that the icon will be switched from a non-display state to a display state, CPU 3 can measure the time since the icon transitioned to a display state.

By the process shown in FIG. 8C, CPU 3 determines whether or not to display an icon of the subject at which the user's gaze is directed, based on the state of the subject. CPU 3 ends the process in FIG. 8C and proceeds to step S8021 in FIG. 8A.

In step S8021 of FIG. 8A, CPU 3 determines whether the result of the display determination of the icon indicating the estimated subject in step S8020 was to "display the icon." If the determination result is to "display the icon," processing proceeds to step S8022. If the determination result is to "hide the icon," processing proceeds to step S8025.

In step S8022, CPU 3 determines whether or not a predetermined time has elapsed since the reference time recorded in step S8314 of FIG. 8C. That is, CPU 3 determines whether or not a predetermined time has elapsed since the time (reference time) when the icon representing the estimated subject transitioned from a non-display state to a display state. If the predetermined time has elapsed since the reference time, processing proceeds to step S8024. If the predetermined time has not elapsed since the reference time, processing proceeds to step S8023.

In step S8023, CPU 3 displays an icon indicating the subject (estimated subject) at which the user's gaze is directed. In step S8024, CPU 3 hides the icon indicating the estimated subject. When the icon indicating the estimated subject becomes displayed and hidden in steps S8023 and S8024, respectively, processing returns to step S8001.

In step S8025, CPU 3 hides the icon indicating the subject (estimated subject) at which the user's gaze is directed, in accordance with the determination result in step S8020. In step S8026, CPU 3 initializes the reference time recorded in step S8207 or step S8314, and returns to step S8001.

Display control of an icon (item) indicating an estimated subject will be described with reference to Fig. 8D and Fig. 8E. In the process of Fig. 8D, the CPU 3 displays an icon indicating an estimated subject at a position based on the user's gaze position. The position based on the user's gaze position includes the position where the user's gaze is detected and the position of the subject detected near the user's gaze position.

The CPU 3 also transitions the icon representing the estimated subject from one of the display state and the non-display state to the other in a transition method selected from a plurality of transition methods. The plurality of transition methods include, for example, a method of gradually changing the size of the icon, a method of gradually changing the transparency of the icon, and a method of immediately transitioning the icon from one of the display state and the non-display state to the other.

Figure 8D is a flowchart of the icon display process according to the first embodiment. The process in Figure 8D is a detailed view of the process in steps S8012 and S8023 in Figure 8A. In the process shown in Figure 8D, the CPU 3 transitions the icon indicating the subject (estimated subject) at which the user's gaze is directed to a display state.

In step S8401, CPU 3 determines whether the icon indicating the estimated subject is currently in a hidden state. If the icon indicating the estimated subject is in a hidden state, processing proceeds to step S8402. If the icon indicating the estimated subject is in a displayed state, the processing shown in FIG. 8D ends.

In step S8402, CPU 3 determines whether or not it was determined in step S8201 of FIG. 8B that "attention is being paid to the subject." If it is determined that attention is being paid to the subject, processing proceeds to S8403. If it is not determined that "attention is being paid to the subject," processing proceeds to S8404.

In step S8403, CPU 3 transitions the icon to a display state by gradually changing the transparency or size of the icon indicating the estimated subject. In a transition method in which the transparency of the icon is gradually decreased, the icon is displayed to become gradually darker. In a transition method in which the size of the icon is gradually increased, the icon is displayed to become gradually larger. By gradually changing the transparency or size of the icon, it becomes easier for the user to recognize the position of the icon indicating the estimated subject.

Figures 15(a) to 15(d) are diagrams showing an example of changing the transparency of an icon to a display state. Figures 16(a) to 16(d) are diagrams showing an example of changing the size of an icon to a display state.

In FIG. 15(a), gaze position 1502 is detected with respect to subject 1501. In FIG. 15(a), the icon indicating gaze position 1502 is hidden. In the situation of FIG. 15(a), when it is determined that the user is looking at the subject, CPU 3 starts displaying icon 1503, and gradually decreases the transparency of icon 1503 indicating the estimated subject, making it darker, as shown in FIG. 15(b), FIG. 15(c), and FIG. 15(d). As the user continues to look at the subject, CPU 3 gradually increases the density of icon 1503 (decreases the transparency) in the order of FIG. 15(b), FIG. 15(c), and FIG. 15(d).

In FIG. 16(a), gaze position 1602 is detected for subject 1601. In FIG. 16(a), the icon indicating gaze position 1602 is hidden. In the situation of FIG. 16(a), when it is determined that the user is looking at the subject, CPU 3 starts displaying icon 1603, and gradually increases the size of icon 1603 indicating the estimated subject, as shown in FIG. 16(b), FIG. 16(c), and FIG. 16(d). As the user continues to look at the subject, CPU 3 gradually increases the size of icon 1603 in the order of FIG. 16(b), FIG. 16(c), and FIG. 16(d).

By gradually changing the transparency or size of the icon (icon 1503, icon 1603) indicating the estimated subject to inform the user of its presence, the user can concentrate on shooting without being unintentionally distracted. In addition, by gradually changing the transparency or size of the icon, the user can easily recognize the position of the icon indicating the estimated subject.

If it is not determined in step S8402 that the subject is being watched, then in step S8404, CPU 3 determines whether the subject was determined to be in focus in step S8304 or step S8308 of FIG. 8C. If it is determined to be in focus, processing proceeds to S8406. If it is not determined to be in focus, i.e., if it is determined to be out of focus, processing proceeds to S8405.

In step S8405, CPU 3 transitions the icon indicating the estimated subject to a display state by gradually changing the transparency or size of the icon, as in step S8403. Note that CPU 3 may transition the icon indicating the estimated subject from a non-display state to a display state using different transition methods in steps S8403 and S8405. For example, if CPU 3 transitions the icon to a display state by changing the transparency in step S8403, CPU 3 may transition the icon to a display state by changing the size in step S8405. Conversely, if CPU 3 transitions the icon to a display state by changing the size in step S8403, CPU 3 may transition the icon to a display state by changing the transparency in step S8405.

In step S8406, CPU 3 prioritizes responsiveness and immediately displays an icon indicating the estimated subject.

Figure 8E is a flowchart of the icon non-display process according to the first embodiment. The process in Figure 8E is a detailed view of the processes in steps S8013, S8024, and S8025 in Figure 8A. In the process shown in Figure 8E, the CPU 3 transitions the icon indicating the subject (estimated subject) at which the user's gaze is directed to a non-display state.

In step S8501, CPU 3 determines whether or not the icon indicating the estimated subject is currently in a displayed state. If the icon indicating the estimated subject is in a displayed state, processing proceeds to step S8502. If the icon indicating the estimated subject is in a hidden state, the processing shown in FIG. 8E ends.

In step S8502, CPU 3 determines whether or not a blink of the user was detected in step S8201 of FIG. 8B. That is, CPU 3 determines whether or not the user blinked and the icon would be switched to hidden mode. If a blink is detected, processing proceeds to step S8503. If a blink is not detected, processing proceeds to step S8505.

In step S8503, CPU 3 acquires blinking characteristic information of the user, specifically, information on the time the user's eyes are closed during a blink. CPU 3 can acquire information on the time the user's eyes are closed during a blink, for example, by analyzing the difference between eye images over time using image processing unit 208.

In step S8504, CPU 3 determines a predetermined transition time until the icon representing the estimated subject becomes invisible based on the blinking characteristic information acquired in step S8503. The predetermined transition time is determined to be longer than the time during which the user closes their eyes while blinking. CPU 3 gradually changes the transparency or size of the icon representing the estimated subject over the predetermined transition time, thereby transitioning the icon to the invisible state.

It is assumed that the time for which the eyes are closed while blinking may differ depending on the degree of eye fatigue, or may differ for each user. By transitioning the icon representing the estimated subject to a hidden state over a period of time (a specified transition time) that is longer than the time for which the user's eyes are closed while blinking, the user can recognize that the icon is gradually transitioning from a visible state to a hidden state.

In step S8505, CPU 3 determines whether the composition has changed due to a zoom operation, pan operation, or tilt operation performed in step S8203 of FIG. 8B, or due to a change in the attitude of camera 1 in step S8204. That is, CPU 3 determines whether the icon indicating the estimated subject has been displayed due to a change in the composition. If it is determined that the composition has changed, processing proceeds to step S8506. If it is determined that the composition has not changed, processing proceeds to step S8507.

In step S8506, CPU 3 transitions the icon indicating the estimated subject to a hidden state by gradually changing the transparency or size of the icon. Immediately after changing the composition, the user's gaze position may become unstable. In addition, the user may change the composition again after changing it. By gradually transitioning the icon from a visible state to a hidden state, the user can check the gaze position until it is hidden. This improves the ease of use for the user in determining the composition.

In step S8507, CPU 3 determines whether or not there are multiple subjects detected near the gaze position, similar to step S8303 in FIG. 8C. If there are multiple subjects detected near the gaze position, processing proceeds to step S8508. If there is only one subject detected near the gaze position, processing proceeds to step S8511.

In step S8508, CPU 3 determines whether the subject at the gaze position in the current image (the subject at which the user's gaze is directed, the estimated subject) is the same as the estimated subject in the previous image. That is, CPU 3 determines whether the subject the user is currently looking at is the same as the subject the user looked at in the past. For example, CPU 3 may determine whether the subject present at a position based on the user's current gaze position is the same as the subject that was present at a position based on the user's gaze position in an image (previous frame) captured before the current image (current frame). If the estimated subject is the same as the previous one, processing proceeds to step S8509. If the estimated subject is not the same as the previous one, processing proceeds to step S8510.

In step S8509, CPU 3 transitions the icon representing the estimated subject to a non-display state by gradually changing the transparency of the icon. In step S8510, CPU 3 transitions the icon representing the estimated subject to a non-display state by gradually reducing the size of the icon.

When transitioning the icon indicating the estimated subject from a displayed state to a hidden state, different transition methods are adopted in steps S8509 and S8510, thereby enabling CPU 3 to notify the user that the estimated subject has changed. Note that the transition methods in steps S8509 and S8510 may be different from each other. For example, contrary to the example of FIG. 8E, the transition method in step S8509 may be a method of gradually changing the size, and the transition method in step S8510 may be a method of gradually changing the transparency.

In step S8511, when only one subject is detected and it is determined that the icon should be hidden, CPU 3 prioritizes responsiveness and immediately transitions the icon representing the estimated subject to a hidden state.

According to the above-mentioned embodiment 1, camera 1 controls whether to display an icon indicating a subject (estimated subject) that is estimated to be the user's gaze direction, based on at least one of the image capture situation and the situation of the subject included in the image. By automatically displaying or hiding the icon indicating the estimated subject, camera 1 can ensure visibility according to the situation and improve operability at the same time.

Furthermore, camera 1 transitions the icon indicating the estimated subject from one of a displayed state and a hidden state to the other in a transition method selected from a plurality of transition methods. For example, camera 1 transitions the icon from one of a displayed state and a hidden state to the other by gradually changing the transparency or size of the icon. The gradual change in the icon state allows the user to concentrate on shooting without being unintentionally distracted. Furthermore, the gradual change in the icon state makes it easier for the user to recognize the position of the icon.

Note that the icon display control process described in FIG. 8A shows an example of a process in which an icon display determination based on the imaging situation (FIG. 8B) is made followed by an icon display determination based on the subject situation (FIG. 8C), but the order of determination is not limited to this. The CPU 3 may determine whether or not to display an icon indicating an estimated subject by determining at least any one of the various conditions described as the imaging situation and the various conditions described as the subject situation in any order.

[Embodiment 2]
A second embodiment will be described with reference to Figures 17A, 17B, 18, and 19. In the second embodiment, when switching between a display state and a non-display state of an icon indicating a subject at which a user's gaze is directed (a subject present at a position based on the gaze position, an estimated subject), a transition method is selected based on the state of the subject.

The processes in Figures 8A to 8C are the same as those in the first embodiment. In the second embodiment, the processes shown in Figures 17A and 17B are executed instead of the processes shown in Figures 8D and 8E. The same processes as those in Figures 8D and 8E are denoted by the same reference numerals and detailed descriptions are omitted.

FIG. 17A is a flowchart of the icon display process according to the second embodiment. If it is determined in step S8402 that the subject is being watched, the process proceeds to step S9001. If it is not determined in step S8404 that the estimated subject is in focus, the process proceeds to step S9004.

In step S9001, the icon indicating the estimated subject is not displayed, and the user is paying attention to the estimated subject. The CPU 3 determines whether the distance between the estimated subject (the subject that exists at a position based on the gaze position) and the other subject is smaller than a predetermined threshold. If multiple subjects exist in the image, the other subject can be, for example, the subject that is closest to the estimated subject. The distance between the estimated subject and the other subject can be calculated, for example, using the coordinates of each subject in the image (for example, the coordinates of the center of gravity of each subject). If the distance between the estimated subject and the other subject is smaller than the predetermined threshold and the multiple subjects are close to each other, the process proceeds to step S9002. If the distance between the estimated subject and the other subject is larger than the predetermined threshold and the multiple subjects are not close to each other, the process proceeds to step S9003. Note that if the distance between the estimated subject and the other subject is the predetermined threshold, the CPU 3 may proceed to either step S9002 or step S9003.

In step S9002, where multiple subjects are close to each other, the CPU 3 transitions the icon representing the estimated subject to a display state by gradually changing (increasing) the size of the icon. On the other hand, in step S9003, where multiple subjects are not close to each other, the CPU 3 transitions the icon representing the estimated subject to a display state by gradually changing (decreasing) the transparency of the icon.

Using Figures 18(a) to 18(d), we will explain icon display control when an estimated subject is close to another subject and the distance between the subjects is smaller than a predetermined threshold. Figures 18(a) to 18(d) are diagrams that explain icon display for multiple subjects that are close to each other. In the example of Figure 18(a), CPU 3 estimates, as the estimated subject, subject 1802, which is closer to the head, which is the detection target, from gaze position 1803 (not displayed on display device 10), out of subjects 1801 and 1802.

In FIG. 18(a), subjects 1801 and 1802 are close to each other on the image plane. In FIG. 18(b), CPU 3 starts the display process of icon 1804 indicating the estimated subject for subject 1802, which is the estimated subject (step S9002). In FIG. 18(b), icon 1804 is displayed at a size smaller than the head area of estimated subject 1802. CPU 3 can notify the user of the position of the estimated subject by gradually increasing the size of icon 1804 from the state in FIG. 18(b), as shown in the order of FIG. 18(c) and FIG. 18(d). Note that in FIGS. 18(b) to 18(d), gaze position 1803 is not displayed on display device 10.

Using Figures 19(a) to 19(d), we will explain icon display control when the estimated subject and other subjects are not close to each other and the distance between the subjects is equal to or greater than a predetermined threshold. Figures 19(a) to 19(d) are diagrams that explain icon display for multiple subjects that are not close to each other. In the example of Figure 19(a), CPU 3 estimates, as the estimated subject, subject 1902, which is closer to the head, which is the detection target, from gaze position 1903 (not displayed on display device 10), out of subjects 1901 and 1902.

In FIG. 19(a), subject 1901 and subject 1902 are not close to each other on the image plane. In FIG. 19(b), CPU 3 starts the display process of icon 1904 indicating the estimated subject for subject 1902, which is the estimated subject (step S9003). In FIG. 19(b), icon 1904 is displayed with a density with a higher transmittance than normal display. CPU 3 increases the display density of icon 1904 by gradually lowering the transmittance of icon 1904 from the state in FIG. 19(b) as shown in the order of FIG. 19(c) and FIG. 19(d). This enables CPU 3 to notify the user of the position of the estimated subject. Note that in FIGS. 19(b) to 19(d), gaze position 1903 is not displayed on display device 10.

In step S9004 of FIG. 17A, the icon indicating the estimated subject is not displayed, and the estimated subject has not been determined to be in focus, i.e., the estimated subject is out of focus. As in step S9001, CPU 3 determines whether the distance between the estimated subject and the other subject is smaller than a predetermined threshold. If the distance between the estimated subject and the other subject is smaller than the predetermined threshold, processing proceeds to step S9005. If the distance between the estimated subject and the other subject is equal to or greater than the predetermined threshold, processing proceeds to step S9006.

In step S9005 where the plurality of subjects are close to each other, similar to step S9002, CPU 3 transitions the icon to a display state by gradually changing (increasing) the size of the icon representing the estimated subject. On the other hand, in step S9006 where the plurality of subjects are not close to each other, CPU 3 transitions the icon to a display state by gradually changing (decreasing) the transparency of the icon representing the estimated subject, similar to step S9003.

According to the icon display control described in FIG. 17A, when the estimated subject at which the user's gaze is directed is close to other subjects, the CPU 3 can make the estimated subject easier to recognize by gradually increasing the size of the icon representing the estimated subject.

FIG. 17B is a flowchart of icon non-display processing according to the second embodiment. When a user's blink is detected and blink characteristic information is acquired in step S8503, processing proceeds to step S9011. When it is determined in step S8505 that the composition has been changed, processing proceeds to step S9014.

In step S9011, the icon indicating the estimated subject is in a displayed state, and the icon will be switched to hidden when the user blinks. Similar to step S9001 in FIG. 17A, CPU 3 determines whether the distance between the estimated subject and the other subject is smaller than a predetermined threshold. If the distance between the estimated subject and the other subject is smaller than the predetermined threshold, processing proceeds to step S9012. If the distance between the estimated subject and the other subject is equal to or greater than the predetermined threshold, processing proceeds to step S9013.

In step S9012, where multiple subjects are close to each other, CPU 3 transitions the icon to a non-display state by gradually changing (reducing) the size of the icon representing the estimated subject over a predetermined transition time. On the other hand, in step S9013, where multiple subjects are not close to each other, CPU 3 transitions the icon to a non-display state by gradually changing (increasing) the transparency of the icon representing the estimated subject over a predetermined transition time. The predetermined transition time is determined in the same manner as in step S8504 of FIG. 8E.

In step S9014, the icon indicating the estimated subject is in a displayed state, and after the composition is changed, a predetermined time has elapsed and the icon will be switched to a hidden state. Similar to step S9001 in FIG. 17A, CPU 3 determines whether the distance between the estimated subject and the other subject is smaller than a predetermined threshold. If the distance between the estimated subject and the other subject is smaller than the predetermined threshold, processing proceeds to step S9015. If the distance between the estimated subject and the other subject is equal to or greater than the predetermined threshold, processing proceeds to step S9016.

In step S9015, where multiple subjects are close to each other, CPU 3 transitions the icon representing the estimated subject to a non-display state by gradually changing (reducing) the size of the icon. On the other hand, in step S9016, where multiple subjects are not close to each other, CPU 3 transitions the icon representing the estimated subject to a non-display state by gradually changing (increasing) the transparency of the icon.

According to the above-mentioned embodiment 2, the camera 1 can improve the visibility of the icon and the image by appropriately selecting the transition method between the display state and the non-display state of the icon indicating the estimated subject based on the state of the subject. By appropriately selecting the transition method, the user can easily recognize the icon indicating the estimated subject.

17A and 17B, the subject situation is described as the distance between the estimated subject and other subjects, but is not limited to this. The subject situation may be the number of subjects, the size of the subject, the brightness of the subject (luminance information), etc. The camera 1 can improve the visibility of the icon and image by appropriately selecting a transition method between the display state and the non-display state according to the subject situation.

[Embodiment 3]
20A and 20B, the third embodiment will be described. In the second embodiment, when switching between a display state and a non-display state of an icon indicating a subject (estimated subject) at which the user's gaze is directed, a transition method is selected based on the state of the subject. In contrast, in the third embodiment, the transition method when switching between a display state and a non-display state of an icon indicating an estimated subject is selected based on the zoom state at the time of shooting.

The processes in Figures 8A to 8C are the same as those in the first embodiment. In the third embodiment, the processes shown in Figures 20A and 20B are executed instead of the processes shown in Figures 8D and 8E. The same processes as those in Figures 8D and 8E are denoted by the same reference numerals and detailed descriptions are omitted.

FIG. 20A is a flowchart of the icon display process according to the third embodiment. If it is determined in step S8402 that the subject is being watched, the process proceeds to step S9021. If it is not determined in step S8404 that the estimated subject is in focus, the process proceeds to step S9024.

In step S9021, the icon indicating the estimated subject is not displayed, and the user is focusing on the estimated subject. CPU 3 acquires information on the current zoom state and determines whether or not shooting is occurring on the wide-angle side. CPU 3 can acquire information on whether or not the current zoom state (zoom position) is on the wide-angle side from zoom drive section 120 of photographing lens unit 1A, which controls the zoom state. Also, if the angle of view has been changed by electronic zoom, CPU 3 can determine whether or not shooting is occurring on the wide-angle side based on the electronic zoom setting state on the camera side. If shooting is occurring on the wide-angle side, processing proceeds to step S9022. If shooting is not occurring on the wide-angle side, processing proceeds to step S9023.

In step S9022, where wide-angle shooting is performed, CPU 3 transitions the icon to a display state by gradually changing (lowering) the transparency of the icon indicating the estimated subject. On the other hand, in step S9023, where shooting is not performed on the wide-angle side, CPU 3 transitions the icon to a display state by gradually changing (increasing) the size of the icon indicating the estimated subject. Generally, when shooting with a telephoto zoom state, the subject is perceived as crowded due to a compression effect. For this reason, when shooting on the telephoto side (not wide-angle shooting), CPU 3 gradually increases the size of the icon, making it easier to recognize the estimated subject and improving the visibility of the icon and image.

In step S9024, the icon indicating the estimated subject is not displayed, and the estimated subject has not been determined to be in focus, i.e., the estimated subject is out of focus. As in step S9021, CPU 3 determines whether or not shooting was performed on the wide-angle side. If shooting was performed on the wide-angle side, processing proceeds to step S9025. If shooting was not performed on the wide-angle side, processing proceeds to step S9026.

In step S9025, which is wide-angle shooting, similar to step S9022, CPU 3 transitions the icon to a display state by gradually changing (lowering) the transparency of the icon indicating the estimated subject. On the other hand, in step S9026, which is telephoto shooting, CPU 3 transitions the icon to a display state by gradually changing (increasing) the size of the icon indicating the estimated subject, similar to step S9023.

According to the icon display control described in FIG. 20A, when the subject is viewed as crowded due to the compression effect when shooting at a telephoto setting, the CPU 3 can make the estimated subject easier to recognize by gradually increasing the size of the icon representing the estimated subject.

Figure 20B is a flowchart of icon non-display processing according to embodiment 3. When a user's blink is detected and blink characteristic information is acquired in step S8503, processing proceeds to step S9031. When it is determined in step S8505 that the composition has been changed, processing proceeds to step S9034.

In step S9031, the icon indicating the estimated subject is displayed, and the icon will be hidden when the user blinks. CPU 3 determines whether or not the image was captured on the wide-angle side, similar to step S9021 in FIG. 20A. If the image was captured on the wide-angle side, processing proceeds to step S9032. If the image was not captured on the wide-angle side (if the image was captured on the telephoto side), processing proceeds to step S9033.

In step S9032 when shooting on the wide-angle side, CPU 3 gradually changes (increases) the transparency of the icon indicating the estimated subject over a predetermined transition time, thereby transitioning the icon to a non-display state. On the other hand, in step S9033 when shooting on the telephoto side, CPU 3 gradually changes (reduces) the size of the icon indicating the estimated subject over a predetermined transition time, thereby transitioning the icon to a non-display state. The predetermined transition time is determined in the same manner as in step S8504. When shooting on the telephoto side (not wide-angle side), CPU 3 gradually reduces the size of the icon, making it easier to recognize the estimated subject and improving the visibility of the icon and image.

In step S9034, the icon indicating the estimated subject is in a displayed state, and after the composition is changed, a predetermined time has passed and the icon will be switched to a hidden state. As in step S9031, CPU 3 determines whether or not shooting was performed on the wide-angle side. If shooting was performed on the wide-angle side, processing proceeds to step S9035. If shooting was not performed on the wide-angle side, processing proceeds to step S9036.

In step S9035 when shooting on the wide-angle side, CPU 3 transitions the icon indicating the estimated subject to a non-display state by gradually changing (increasing) the transparency of the icon. On the other hand, in step S9036 when shooting on the telephoto side, CPU 3 transitions the icon indicating the estimated subject to a non-display state by gradually changing (reducing) the size of the icon.

According to the above-mentioned embodiment 3, the camera 1 can improve the visibility of the icon and the image by appropriately selecting a transition method between a display state and a non-display state of the icon indicating the estimated subject based on the zoom state information at the time of shooting. By appropriately selecting the transition method, the user can easily recognize the icon indicating the estimated subject.

In the above embodiments, an example of selecting a transition method for transitioning an icon indicating an estimated subject from one of a display state and a non-display state to the other based on various conditions has been described, but the camera 1 may select a transition method based on conditions other than the above. For example, the camera 1 may select a transition method based on the movement of the subject (information on the movement characteristics), such as whether the subject present at a position based on the user's gaze position is stationary or moving while accelerating. For example, when a subject moving while accelerating stops, the camera 1 gradually changes (increases) the transmittance of the icon indicating the estimated subject, thereby transitioning the icon to a non-display state. The camera 1 may also select a transition method based on whether the detection of the user's gaze position has succeeded or failed. Furthermore, the camera 1 may select a transition method based on a condition that is an arbitrary combination of these conditions. The transition method to be selected based on various conditions can be set in advance in consideration of the visibility of the icon and the screen.

The above embodiment is merely an example, and configurations obtained by appropriately modifying or changing the configuration of the above embodiment within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the above embodiments are also included in the present invention.

For example, although an example of viewing a subject through an EVF (electronic viewfinder) has been described, the present invention can also be applied when viewing a subject through an OVF (optical viewfinder). The present invention can be applied to devices other than imaging devices (cameras), and can also be applied when viewing three-dimensional computer graphics, etc. The present invention can be applied to various electronic devices that can detect the viewpoint.

[Examples of application to other electronic devices]
Fig. 21(a) is an external view of a notebook personal computer 2110 (notebook PC). In Fig. 21(a), an imaging unit 2115 that captures an image of a user looking at a display unit 2111 of the notebook PC 2110 is connected to the notebook PC 2110, and the notebook PC 2110 obtains the imaging results from the imaging unit 2115. The notebook PC 2110 then detects the user's viewpoint based on the imaging results. The present invention is also applicable to the notebook PC 2110.

Fig. 21(b) is an external view of the smartphone 2120. In Fig. 21(b), the smartphone 2120 detects the viewpoint of a user looking at the display unit 2122 of the smartphone 2120 based on the imaging results of the in-camera 2121 (front camera). The present invention is also applicable to the smartphone 2120. Similarly, the present invention is also applicable to various tablet terminals.

Figure 21 (c) is an external view of the game console 2130. In Figure 21 (c), a head mounted display 2135 (HMD) that displays a VR (Virtual Reality) image of the game on a display unit 2136 is connected to the game console 2130. The HMD 2135 has a camera 2137 that captures an image of the eye of a user wearing the HMD 2135, and the game console 2130 acquires the image capture result from the HMD 2135. The game console 2130 then detects the user's viewpoint based on the image capture result. The present invention is also applicable to the game console 2130. The components of the game console 2130 may be built into the HMD 2135. Therefore, the present invention is also applicable to the HMD 2135. Just as the present invention can be applied when viewing a VR image displayed on an HMD, the present invention can also be applied when viewing an AR (Augmented Reality) image displayed on the lens portion of a glasses-type wearable device, or when viewing an actual object beyond the lens portion. Just as the present invention can be applied to VR technology and AR technology, the present invention can also be applied to other xR technologies such as MR (Mixed Reality) technology and SR (Substitutional Reality) technology.

[Other embodiments]
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The disclosure of the present embodiment includes the following configuration, method, program, and medium.
(Configuration 1)
An acquisition means for acquiring information on a gaze position of a user viewing a captured image;
a control means for controlling the display of an item at a position based on the user's line of sight;
The electronic device according to claim 1, wherein the control means transitions the item from one of a displayed state and a non-displayed state to the other in a transition manner selected from a plurality of transition manners.
(Configuration 2)
The plurality of transition methods include:
a transition scheme in which the item transitions from one of a visible state and a hidden state to the other by gradually changing the size of the item;
a transition scheme in which the item is transitioned from one of a visible state and a hidden state to the other by gradually changing the transparency of the item;
2. The electronic device according to configuration 1, further comprising at least one of a display state and a non-display state for immediately transitioning the item from one of the display state and the non-display state to the other.
(Configuration 3)
The method further includes estimating an object at which the user's gaze is directed, among objects included in the image, based on the user's gaze position,
3. The electronic device according to configuration 1 or 2, wherein the control means controls so that the item is displayed at the position of the subject estimated by the estimation means.
(Configuration 4)
The electronic device according to any one of configurations 1 to 3, characterized in that the control means selects a transition method for transitioning the item to a display state based on whether the user is paying attention to the image.
(Configuration 5)
The electronic device according to any one of configurations 1 to 4, wherein the control means selects a transition method for transitioning the item to a hidden state based on whether or not the user has blinked.
(Configuration 6)
The electronic device described in configuration 5, characterized in that when the user blinks, the control means transitions the item from a displayed state to a hidden state by gradually changing the state of the item over a period of time longer than a previously measured period of time the user's eyes are closed while blinking.
(Configuration 7)
The electronic device according to any one of configurations 1 to 6, characterized in that the control means selects a transition method for transitioning the item to a hidden state based on whether the composition of the image has been changed by a zoom operation, a pan operation, or a tilt operation.
(Configuration 8)
The electronic device according to any one of configurations 1 to 7, characterized in that the control means selects a transition method for transitioning the item to a hidden state based on whether or not a composition of the image has changed due to a change in the attitude of an imaging device that captures the image.
(Configuration 9)
The electronic device according to any one of configurations 1 to 8, characterized in that the control means selects a transition method for transitioning the item to a display state based on whether a subject present at a position based on the user's gaze position is in-focus or out-of-focus.
(Configuration 10)
The electronic device according to any one of configurations 1 to 9, wherein the control means selects a transition method for transitioning the item to a hidden state based on whether or not the image includes multiple subjects.
(Configuration 11)
The electronic device of any one of configurations 1 to 10, characterized in that the control means transitions the item to a hidden state using different transition methods depending on whether the subject currently being viewed by the user is the same as or different from a subject previously viewed by the user, when the image contains a plurality of subjects.
(Configuration 12)
12. The electronic device according to configuration 10 or 11, wherein the subject included in the image is a subject of a predetermined type.
(Configuration 13)
The electronic device according to any one of configurations 10 to 12, wherein the number of subjects included in the image is the number of subjects included in a predetermined range including the user's line of sight.
(Configuration 14)
The electronic device described in configuration 13, characterized in that the specified range is determined based on at least one of the focal length, shooting distance, shooting mode, and size of a subject present at a position based on the user's line of sight when capturing the image.
(Configuration 15)
The electronic device of any one of configurations 1 to 14, characterized in that the control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other based on a distance between a subject present at a position based on the user's gaze position and another subject.
(Configuration 16)
The electronic device described in configuration 15, characterized in that the control means transitions the item from one of a displayed state and a hidden state to the other using different transition methods when the distance between a subject present at a position based on the user's gaze position and another subject is smaller than a predetermined threshold and when the distance is greater than the predetermined threshold.
(Configuration 17)
The electronic device according to any one of configurations 1 to 16, characterized in that the control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other based on a zoom state when capturing the image.
(Configuration 18)
The electronic device described in configuration 17, characterized in that the control means transitions the item from one of the displayed state and the hidden state to the other using different transition methods when the zoom state is on the wide-angle side and when it is on the telephoto side.
(Configuration 19)
The electronic device of any one of configurations 1 to 18, characterized in that the control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other based on the movement of a subject present at a position based on the user's gaze position.
(Configuration 20)
The electronic device of any one of configurations 1 to 19, characterized in that the control means transitions the item from one of a displayed state and a hidden state to the other by different transition methods depending on whether detection of the user's gaze position is successful or unsuccessful.
(method)
An acquisition step of acquiring information on a gaze position of a user viewing a captured image;
and a control step of controlling the display of the item at a position based on the user's gaze position,
The control method for an electronic device, wherein the control step transitions the item from one of a displayed state and a non-displayed state to the other in a transition manner selected from a plurality of transition manners.
(program)
A program for causing a computer to function as each of the means of the electronic device according to any one of configurations 1 to 20.
(Medium)
A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device described in any one of configurations 1 to 20.

1: Camera (electronic device), 3: CPU, 10: Display device, 11: Display device drive circuit, 17: Eye image sensor, 201: Line of sight detection circuit, 208: Image processing unit

Claims (23)

An acquisition means for acquiring information on a gaze position of a user viewing a captured image;
a control means for controlling the display of an item at a position based on the user's line of sight;
The electronic device according to claim 1, wherein the control means transitions the item from one of a displayed state and a non-displayed state to the other in a transition manner selected from a plurality of transition manners. The plurality of transition methods include:
a transition scheme in which the item transitions from one of a visible state and a hidden state to the other by gradually changing the size of the item;
a transition scheme in which the item is transitioned from one of a visible state and a hidden state to the other by gradually changing the transparency of the item;
2. The electronic device according to claim 1, further comprising at least one of a display state and a non-display state, and a transition method for immediately transitioning the item from one of the display state and the non-display state to the other. The method further includes estimating an object at which the user's gaze is directed, among objects included in the image, based on the user's gaze position,
The electronic device according to claim 1 , wherein the control means performs control so as to display the item at the position of the subject estimated by the estimation means. The electronic device according to claim 1 , wherein the control means selects a transition method for transitioning the item to a display state based on whether the user is paying attention to the image. The electronic device according to claim 1 , wherein the control means selects a transition method for transitioning the item to a non-display state based on whether or not the user has blinked. The electronic device according to claim 5, characterized in that, when the user blinks, the control means transitions the item from a displayed state to a hidden state by gradually changing the state of the item over a period of time longer than a previously measured period of time that the user's eyes are closed during a blink. The electronic device according to claim 1 , wherein the control means selects a transition method for transitioning the item to a hidden state based on whether a composition of the image has been changed by a zoom operation, a pan operation, or a tilt operation. The electronic device according to claim 1, characterized in that the control means selects a transition method for transitioning the item to a hidden state based on whether or not a composition of the image has changed due to a change in the attitude of an imaging device that captures the image. The electronic device according to claim 1, characterized in that the control means selects a transition method for transitioning the item to a display state based on whether a subject present at a position based on the user's gaze position is in-focus or out-of-focus. The electronic device according to claim 1 , wherein the control means selects a transition method for transitioning the item to a non-display state based on whether the image includes a plurality of subjects. The electronic device according to claim 1, characterized in that the control means transitions the item to a hidden state using different transition methods depending on whether the image contains multiple objects and the object currently being viewed by the user is the same as or different from the object previously viewed by the user. The electronic device according to claim 10 , wherein the subject included in the image is a subject of a predetermined type. The electronic device according to claim 10 , wherein the number of subjects included in the image is the number of subjects included in a predetermined range including the user's line of sight. 14. The electronic device according to claim 13, wherein the predetermined range is determined based on at least one of a focal length, a shooting distance, a shooting mode, and a size of a subject present at a position based on a line of sight of the user when capturing the image. The electronic device according to claim 1, characterized in that the control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other based on a distance between a subject present at a position based on the user's gaze position and another subject. The electronic device described in claim 15, characterized in that the control means transitions the item from one of the displayed state and the hidden state to the other using different transition methods when the distance between the subject present at a position based on the user's gaze position and another subject is smaller than a predetermined threshold and when the distance is greater than the predetermined threshold. The electronic device according to claim 1 , wherein the control means selects a transition method for transitioning the item from one of a displayed state and a non-displayed state to the other, based on a zoom state when the image is captured. 18. The electronic device according to claim 17, wherein the control means transitions the item from one of the displayed state and the non-displayed state to the other by different transition methods depending on whether the zoom state is on the wide-angle side or the telephoto side. The electronic device according to claim 1 , characterized in that the control means selects a transition method for transitioning the item from one of a displayed state and a hidden state to the other based on the movement of a subject present at a position based on the user's gaze position. The electronic device according to claim 1, characterized in that the control means transitions the item from one of a displayed state and a hidden state to the other using different transition methods depending on whether detection of the user's gaze position is successful or unsuccessful. An acquisition step of acquiring information on a gaze position of a user viewing a captured image;
and a control step of controlling the display of the item at a position based on the user's gaze position,
The control method for an electronic device, wherein the control step transitions the item from one of a displayed state and a non-displayed state to the other in a transition manner selected from a plurality of transition manners. A program for causing a computer to function as each of the means of the electronic device described in any one of claims 1 to 20. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 20.

Images