JP2023004678A - Processing device and control method therefor - Google Patents

Abstract

To achieve stability and quick responsiveness in visual line detection.SOLUTION: With a Kalman filter using a low-order polynomial regression, a system control unit 50 corrects visual line information corresponding to a position of a visual line of a person.SELECTED DRAWING: Figure 6

Description

The present invention relates to a device for acquiring a position corresponding to a person's line of sight.

A method for detecting the line-of-sight position of a person is known. Patent Literature 1 discloses a method of detecting the line-of-sight position of a subject by illuminating the cornea and photographing its reflected image. Further, Patent Document 1 discloses means for predicting the current line-of-sight position using a Kalman filter from line-of-sight positions obtained at a plurality of times in the past.

JP 2018-197974 A

However, Patent Document 1 does not disclose in detail the Kalman filter for predicting the line-of-sight position. SUMMARY OF THE INVENTION An object of the present invention is to derive a line-of-sight position in consideration of a balance between responsiveness and stability of line-of-sight detection in an imaging apparatus having a line-of-sight position detecting means and a line-of-sight display means.

The present invention includes acquisition means for acquiring line-of-sight information corresponding to the line-of-sight position of a person, and correction means for correcting the line-of-sight information, wherein the correction means is a Kalman filter using a low-order polynomial regression equation. is configured to correct the line-of-sight information by

According to the present invention, it is possible to achieve both stability and responsiveness in line-of-sight detection.

1 is a block diagram showing the configuration of an imaging device according to an embodiment of the present invention; FIG. FIG. 2 is a diagram showing a correspondence relationship between a pupil plane of pixels and a photoelectric conversion unit of an imaging device according to an embodiment of the present invention; FIG. 4 is a diagram showing the correspondence relationship between the pupil plane of the pixel and the aperture of the imaging device according to the embodiment of the present invention; The figure which shows the structure of the gaze input operation part concerning embodiment of this invention. 4 is a flow chart of focus detection, line-of-sight detection, and shooting operation of the imaging device according to the embodiment of the present invention; 1 is a flow chart of a line-of-sight prediction method for an imaging device according to an embodiment of the present invention; Schematic diagram of a line-of-sight prediction method for an imaging device according to an embodiment of the present invention.

Preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[Description of configuration of imaging device]
FIG. 1 is a block diagram showing the configuration of an imaging device according to an embodiment of the present invention. In FIG. 1, a lens unit 150 is a lens unit that mounts an interchangeable photographing lens. Although the lens 103 is normally composed of a plurality of lenses, only one lens is shown here for simplicity. A communication terminal 6 is a communication terminal for the lens unit 150 to communicate with the digital camera 100 side, and a communication terminal 10 is a communication terminal for the digital camera 100 to communicate with the lens unit 150 side. The lens unit 150 communicates with the system control section 50 via the communication terminals 6 and 10, controls the diaphragm 102 via the diaphragm driving circuit 2 by the internal lens system control circuit 4, and controls the diaphragm 102 via the AF driving circuit 3. The focus is adjusted by displacing the position of the lens 103.

The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50 . The imaging unit 22 is an imaging device configured by a CCD, a CMOS device, or the like that converts an optical image into an electrical signal. The A/D converter 23 converts analog signals into digital signals. The A/D converter 23 is used to convert the analog signal output from the imaging section 22 into a digital signal. Signals obtained from the imaging unit 22 are used not only for imaging but also for exposure control and focus detection control. The imaging unit 22 is provided with pixels obtained by dividing the photoelectric conversion unit for one microlens. By dividing the photoelectric conversion unit, the entrance pupil is divided, and a phase difference detection signal can be obtained from each photoelectric conversion unit. Further, by adding the signals from the divided photoelectric conversion units, an imaging signal can also be obtained.

Such pixels have the advantage that they can be used both as focus detection pixels and imaging pixels.

FIG. 2 shows the configuration of a pixel according to this embodiment and the correspondence relationship between the pupil plane and the photoelectric conversion unit. 201 denotes a photoelectric conversion unit, 253 denotes a pupil plane, 251 denotes a microlens, and 252 denotes a color filter. In FIG. 2, two photoelectric conversion units 201 are provided: a photoelectric conversion unit 201a (first focus detection pixel) and a photoelectric conversion unit 201b (second focus detection pixel). In the photoelectric conversion unit 201a, light passing through the pupil plane indicated by 253a enters the photoelectric conversion unit 201a. Also, in the photoelectric conversion unit 201b, the light passing through the pupil plane indicated by 253b enters the photoelectric conversion unit 201b. Accordingly, focus detection can be performed from the signals obtained from the photoelectric conversion units 201a and 201b. Further, by adding the signals obtained from the photoelectric conversion units 201a and 201b, an imaging signal can be generated.

In this embodiment, by providing the pixels shown in FIG. 2 in the entire screen area of the imaging unit 22, any subject captured on the screen can be focused by phase difference detection.

In this embodiment, the above focus detection method will be described, but the focus detection method is not limited to this case. For example, the imaging unit 22 may be provided with focus detection pixels shown in FIG. 3, which will be described later, to perform focus detection. Alternatively, the imaging unit 22 may be provided with only imaging pixels instead of pixels for focus detection, and focus detection may be performed by a contrast method.

FIG. 3 shows the configuration of the focus detection dedicated pixels and the correspondence relationship between the pupil plane and the photoelectric conversion unit. Unlike FIG. 2, FIG. 3 shows pixels dedicated to focus detection. The shape of pupil plane 253 is determined by aperture 254 . In addition, since only the light that has passed through the pupil plane 253 is detected, it is necessary to separately provide a pair of pixels, pixels for detecting light from the right pupil plane (not shown in FIG. 3), and to acquire the focus detection signal. be. By providing the focus detection pixels shown in FIG. 3 and the imaging pixels in the entire screen area of the imaging unit 22, it is possible to focus on any subject captured on the screen by phase difference detection.

The image processing unit 24 performs resizing processing such as predetermined pixel interpolation and reduction, and color conversion processing on the data from the A/D converter 23 or the data from the memory control unit 15 . Further, the image processing unit 24 performs predetermined arithmetic processing using captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained arithmetic results. As a result, TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing are performed. The image processing unit 24 further performs predetermined arithmetic processing using the captured image data, and also performs TTL AWB (Auto White Balance) processing based on the obtained arithmetic results.

Output data from the A/D converter 23 is directly written into the memory 32 via the image processing section 24 and the memory control section 15 or via the memory control section 15 . The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A/D converter 23, and image data to be displayed on the display unit 28 as display means. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, moving images for a predetermined period of time, and audio.

The memory 32 also serves as an image display memory (video memory). The D/A converter 19 converts image display data stored in the memory 32 into an analog signal and supplies the analog signal to the display unit 28 . Thus, the image data for display written in the memory 32 is displayed by the display section 28 via the D/A converter 19 . The display unit 28 displays on a display such as an LCD in accordance with the analog signal from the D/A converter 19 . The digital signal that is once A/D converted by the A/D converter 23 and stored in the memory 32 is converted to analog by the D/A converter 19, and is sequentially transferred to the display unit 28 for display. It functions and enables through image display (live view display). Note that the display unit 28 may be provided with an electronic viewfinder for viewing through an eyepiece (not shown), or may be provided with a display on the rear surface of the digital camera 100 . Also, both an electronic viewfinder and a rear display may be provided.

The nonvolatile memory 56 is an electrically erasable/recordable memory, and for example, an EEPROM or the like is used. The nonvolatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50 . The program here is a program for executing various flowcharts described later in this embodiment.

A system control unit 50 controls the entire digital camera 100 . By executing the program recorded in the non-volatile memory 56 described above, each process of this embodiment, which will be described later, is realized. A system memory 52 uses a RAM. In the system memory 52, constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like are developed. Further, the system control unit also performs display control by controlling the memory 32, the D/A converter 19, the display unit 28, and the like.

A system timer 53 is a timer that measures the time used for various controls and the time of a built-in clock.

A power switch 72 is an operation member for switching ON and OFF of the power of the digital camera 100 .

A mode changeover switch 60 , a first shutter switch 62 , a second shutter switch 64 , and an operation section 70 are operation means for inputting various operation instructions to the system control section 50 .

The mode switch 60 switches the operation mode of the system control unit 50 between a still image recording mode, a moving image shooting mode, a playback mode, and the like. Modes included in the still image recording mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), and a shutter speed priority mode (Tv mode). In addition, there are various scene modes, program AE modes, custom modes, etc., which are shooting settings for each shooting scene. A mode switch 60 switches directly to any of these modes contained in the menu button. Alternatively, after switching to the menu button once with the mode switching switch 60, any of these modes included in the menu button may be switched using another operation member. Similarly, the movie shooting mode may also include multiple modes. The first shutter switch 62 is turned on when the shutter button 61 provided on the digital camera 100 is pressed halfway (imaging preparation instruction), and generates a first shutter switch signal SW1. Operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (automatic white balance) processing, and EF (flash pre-emission) processing are started by the first shutter switch signal SW1.

The second shutter switch 64 is turned ON when the operation of the shutter button 61 is completed, that is, when the shutter button 61 is fully pressed (imaging instruction), and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of photographing processing operations from reading out signals from the imaging unit 22 to writing image data in the recording medium 200. FIG.

Each operation member of the operation unit 70 is appropriately assigned a function for each scene by selecting and operating various function icons displayed on the display unit 28, and acts as various function buttons. The function buttons include, for example, an end button, a return button, an image forward button, a jump button, a refinement button, an attribute change button, and the like. For example, when the menu button is pressed, a menu screen on which various settings can be made is displayed on the display unit 28 . The user can intuitively perform various settings by using the menu screen displayed on the display unit 28, the up, down, left, and right four-direction buttons and the SET button.

The operation unit 70 is various operation members as an input unit that receives operations from the user. The operation unit 70 is provided with electronic buttons, a cross key, and the like for performing menu selection, mode selection, playback of captured moving images, and the like.

In this embodiment, a line-of-sight input operation unit 701 is provided as one of the operation units 70 . The line-of-sight input operation unit 701 is an operation member for detecting which part of the display unit 28 the user's line of sight is looking at.

FIG. 4A shows an example of the line-of-sight input operation unit 701. FIG. FIG. 4A shows a configuration for realizing a method of detecting the rotation angle of the optical axis of the user's eyeball 501a looking into the viewfinder field disclosed in Patent Document 1 and detecting the user's line of sight from the detected rotation angle. is. A live view display image captured through the lens unit 100 is displayed on the display unit 28 . 701a denotes an image sensor, 701b denotes a light receiving lens, 701c denotes a dichroic mirror, 701d denotes an eyepiece lens, and 701e denotes an illumination light source. Infrared light is projected onto the eyeball 501a by the illumination light source 701e. The infrared light reflected by the eyeball 501a is reflected by the dichroic mirror 701c and captured by the image sensor 701a. The photographed eyeball image is converted into a digital signal by an A/D converter (not shown) and transmitted to the system control unit 50 . The system control unit 50 serving as line-of-sight information generation means and line-of-sight position information output means extracts the pupil region and the like from the photographed eyeball image, and calculates the user's line of sight.

Note that the line-of-sight input operation unit 701 is not limited to this method, and may adopt a method of photographing both eyes of the user and detecting the line of sight. FIG. 4B shows an example of the line-of-sight input operation section 701 different from that in FIG. In FIG. 4B, a live view display image captured through the lens unit 100 is displayed on the display section 28 provided on the rear surface of the digital camera 100. FIG. In FIG. 4B, a camera 701f is provided on the rear surface of the digital camera 100 to photograph the face 500 of the user observing the display section . In FIG. 5, the dotted line indicates the angle of view captured by the camera 701f. An illumination light source 701e (not shown) projects light onto the user's face, and an eyeball image is acquired by the camera 701f. Thereby, the line of sight of the user is calculated. Note that the line-of-sight input operation unit 701 is not limited to this method, and may have any configuration as long as it can detect which part of the display unit 28 the user is gazing at.

The power control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching blocks to be energized, and the like, and detects whether or not a battery is installed, the type of battery, and the remaining amount of the battery. Also, the power supply control unit 80 controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies necessary voltage to each unit including the recording medium 200 for a necessary period.

The power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. A recording medium I/F 18 is an interface with a recording medium 200 such as a memory card or hard disk. A recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

The communication unit 54 is connected wirelessly or via a priority cable to transmit and receive video signals and audio signals. The communication unit 54 can be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can transmit images (including through images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can receive image data and other various information from external devices. can.

The orientation detection unit 55 detects the orientation of the digital camera 100 with respect to the direction of gravity. Based on the posture detected by the posture detection unit 55, whether the image captured by the imaging unit 22 is an image captured with the digital camera 100 held horizontally or an image captured with the digital camera 100 held vertically is determined. It is identifiable. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the image captured by the imaging unit 22, and rotate and record the image. An acceleration sensor, a gyro sensor, or the like can be used as the posture detection unit 55 .

The digital camera 100 described above is capable of photographing using center single-point AF or face AF. Central one-point AF is to perform AF on one central point within the photographing screen. Face AF is to perform AF on a face in a photographing screen detected by a face detection function.

The face detection function will be explained. The system control unit 50 sends image data for face detection to the image processing unit 24 . Under the control of the system control unit 50, the image processing unit 24 applies a horizontal bandpass filter to the image data. Also, under the control of the system controller 50, the image processor 24 applies a vertical bandpass filter to the processed image data. Edge components are detected from the image data by these horizontal and vertical bandpass filters.

After that, the system control unit 50 performs pattern matching on the detected edge components to extract a group of candidates for eyes, nose, mouth, and ears. Then, the system control unit 50 determines eye pairs that satisfy preset conditions (for example, the distance and inclination of the two eyes) from the group of extracted eye candidates. Only those that have are narrowed down as the candidate group of eyes. Then, the system control unit 50 associates the group of narrowed-down eye candidates with the corresponding other parts (nose, mouth, ears) forming the face, and passes them through a preset non-face condition filter. , to detect faces. The system control unit 50 outputs the face information according to the face detection result, and ends the process. At this time, the feature quantity such as the number of faces is stored in the system memory 52 . The method of realizing the face detection function is not limited to the above-described method, and the number, size, parts, etc. of faces may be similarly detected by a method using known machine learning. Further, the type of subject is not limited to a person's face, and animals, vehicles, and the like may be detected.

As described above, it is possible to perform image analysis on image data displayed in live view display or playback display, extract feature amounts of the image data, and detect subject information. In this embodiment, face information is taken as an example of subject information, but there are various types of subject information such as red-eye determination, eye detection, blink detection, smile detection, and the like.

Note that face AE, face FE, and face WB can be performed simultaneously with face AF. Face AE is to optimize the exposure of the entire screen according to the brightness of the detected face. Face FE is to adjust the light of the flash centering on the detected face. Face WB is to optimize the WB of the entire screen according to the color of the detected face.

[Task]
The following problem arises in an imaging apparatus having means for detecting the line-of-sight position and means for displaying the line-of-sight. In the imaging apparatus, in order for the photographer to look at the display means and comfortably select a subject or a range-finding point based on the line of sight, it is necessary to achieve both stability and responsiveness in displaying the detected line-of-sight position. . The line-of-sight position vibrates and fluctuates due to line-of-sight detection errors and human line-of-sight slight movements. When the detected line-of-sight position is displayed as a pointer on the display device at a rate of several tens of frames per second, visually recognizing the pointer makes the user feel uncomfortable, or makes it difficult to select a subject or a range-finding point. put away. Therefore, in order to ensure the stability of the line of sight, it is necessary to perform filtering using a plurality of past line of sight positions. However, if the current and future line-of-sight positions are predicted using a plurality of line-of-sight positions acquired in the past, a delay occurs due to the phase delay characteristic of the filter, which impairs responsiveness. This delay increases as more past line-of-sight position data is used. If the delay is large when tracking a fast-moving object with the line of sight on the display device, it becomes difficult to catch the object with the line of sight. On the other hand, if the strength of the filter is reduced in order to ensure responsiveness, the line-of-sight vibration cannot be completely removed, resulting in a stability problem. In this way, stability and responsiveness are in principle in a trade-off relationship. In the imaging device, it is conceivable to adjust the filter strength to ensure a balance between stability and responsiveness, but since the accuracy of line-of-sight detection varies depending on the photographer and shooting conditions, the filter strength is set as a fixed value. With such means, it is difficult to achieve both stability and responsiveness. The present embodiment focuses on such technical problems, and balances the stability and responsiveness of line-of-sight detection at the same time.

[Description of line-of-sight detection and shooting operation]
Hereinafter, the sight line position detection processing method according to the first embodiment of the present invention will be described with reference to FIG. FIG. 5 is a flowchart for explaining focus detection, line-of-sight detection, and photographing operations of the imaging apparatus of this embodiment. FIG. 6 shows the operation at the time of live view shooting, in which shooting is performed from a live view state (moving image shooting state) such as a shooting standby state, and is realized mainly by the system control unit 50 .

In S1, under the control of the system control unit 50, the imaging unit 22 is driven to acquire imaging data. Since the imaging data to be acquired is not for recording, which will be described later, but for detection and display, an image smaller in size than the recorded image is acquired. In S1, an image having sufficient resolution for focus detection, subject detection, or live view display is acquired. Here, since the drive operation is for moving image shooting for live view display, shooting is performed using a so-called electronic shutter that performs charge accumulation and readout for a time corresponding to the frame rate for live view display. The live view display performed here is for the photographer to check the shooting range and shooting conditions. can be

In S2, the system control unit 50 acquires the focus detection data obtained from the first focus detection pixel and the second focus detection pixel included in the focus detection area among the imaging data obtained in S1. In addition, the system control unit 50 adds the output signals of the first focus detection pixel and the second focus detection pixel to generate an imaging signal, and acquires image data obtained by applying color interpolation processing in the image processing unit 24. do. In this way, image data and focus detection data can be acquired by one shot. Note that when the imaging pixels, the first focus detection pixels, and the second focus detection pixels are configured as individual pixels, image data is acquired by performing a complementing process for the focus detection pixels.

In S<b>3 , the system control unit 50 uses the image processing unit 24 to generate an image for live view display based on the image data obtained in S<b>2 , and displays the image on the display unit 28 . The image for live view display is, for example, a reduced image that matches the resolution of the display unit 28, and the image processing unit 24 can perform reduction processing when generating image data in S2. In this case, the system control unit 50 causes the display unit 28 to display the image data obtained in S2. As described above, shooting and display are performed at a predetermined frame rate during live view display, so the photographer can adjust the composition and exposure conditions at the time of shooting through the display unit 28 . Further, as described above, in this embodiment, it is possible to detect a person's face, an animal, or the like as a subject. In step S3, when the live view display is started, a frame or the like indicating the area of the detected subject is also displayed.

In S4, the system control unit 50 starts line-of-sight detection and focus detection. After S4, the line-of-sight input operation unit 701 associates which position the photographer is observing on the display unit 28 (line-of-sight position) with the display image observed by the photographer, and displays the position at predetermined time intervals. get. Also, the detected line-of-sight position is displayed on the display unit 28 in order to notify the photographer.

In S5, the system control unit 50 detects ON/OFF of the first shutter switch 62 (Sw1) indicating the start of shooting preparation. The shutter button 61, which is one of the operation units 70, can detect two stages of on/off depending on the amount of depression. It corresponds to ON/OFF of the first step.

If Sw1 is not detected to be on (or is detected to be off) in S5, the system control unit 50 advances the process to S11 and determines whether or not the main switch included in the operation unit 70 is turned off. On the other hand, when Sw1 is detected to be ON in S5, the system control unit 50 advances the process to S6 to set a focus detection area to be focused and perform focus detection. Here, the focus detection area is set using both the line-of-sight position detected in S4 and the subject detection position inside the imaging apparatus. The line-of-sight position detected in S4 has an error with respect to the position of the subject intended by the photographer due to various factors. In addition, although there are individual differences, there is a delay time of about several tenths of a second until the human eye starts to move after recognizing it. In the present invention, the reliability of the detected line-of-sight position information is evaluated according to the photographing conditions, and the line-of-sight information is processed. It is possible to obtain more accurate line-of-sight position information. Details will be described later. In S6, the focus detection area is set using both the line-of-sight position information processed to be described later and the subject detection position inside the imaging apparatus. After S6, the setting of the focus detection area using the line-of-sight position information and the focus detection process are repeatedly executed each time an image is captured.

Using the focus detection data corresponding to the set focus detection area, the defocus amount and direction are obtained for each focus detection area. In this embodiment, the system control unit 50 generates an image signal for focus detection, calculates the shift amount (phase difference) of the focus detection signal, and obtains the defocus amount and direction from the calculated shift amount. It shall be.

The first focus detection signal and the second focus detection signal obtained as image signals for focus detection from the set focus detection area are subjected to shading correction and filtering to reduce the light amount difference between the pair of signals and to reduce the phase difference. Perform signal extraction of the spatial frequency to be detected. Next, shift processing is performed to relatively shift the filtered first focus detection signal and the second focus detection signal in the direction of pupil division, and a correlation amount representing the degree of matching between the signals is calculated.

Let A(k) be the k-th first focus detection signal after filtering, B(k) be the second focus detection signal, and W be the range of number k corresponding to the focus detection area. Furthermore, when the shift amount by the shift process is s1 and the shift range of the shift amount s1 is Γ1, the correlation amount COR is calculated by Equation (1).

By the shift processing of the shift amount s1, the k-th first focus detection signal A(k) and the k-s1-th second focus detection signal B(k-s1) are correlated and subtracted to generate a shift subtraction signal. The absolute value of the generated shift subtraction signal is calculated, the sum of the numbers k is taken within the range W corresponding to the focus detection area, and the correlation amount COR(s1) is calculated. If necessary, the correlation amount calculated for each row may be added over a plurality of rows for each shift amount.

Next, from the correlation amount, a sub-pixel calculation is performed to calculate the real-value shift amount that minimizes the correlation amount, and this is used as the image shift amount p1. Then, the calculated image shift amount p1 is multiplied by a conversion coefficient K1 corresponding to the image height of the focus detection area, the F value of the imaging lens (imaging optical system), and the exit pupil distance to detect the detected defocus amount. .

In S7, the system control unit 50 drives the lens based on the defocus amount detected in the selected focus detection area. If the detected defocus amount is smaller than the predetermined value, it is not necessary to drive the lens.

Next, in S8, acquisition of an image for detection/display and live view display performed in S1 and focus detection processing performed in S6 are performed. On the live view display, information on the subject area and line-of-sight position detected as described above is also superimposed and displayed. The processing performed in S8 may be performed in parallel while driving the lens in S7. Also, the focus detection area may be changed in accordance with the obtained line-of-sight position in accordance with the live view display that is updated as needed. When the focus detection process is finished, the process proceeds to S9, and the system control unit 50 detects ON/OFF of the second shutter switch 64 (Sw2) that indicates an instruction to start photographing. A release (shooting trigger) switch, which is one of the operation units 70, can detect two stages of on/off depending on the amount of depression. Equivalent to turning on/off the eyes. If the ON state of Sw2 is not detected in S9, the system control unit 50 returns to S5 and detects the ON/OFF state of Sw1.

When the ON state of Sw2 is detected in S9, the system control unit 50 advances the process to S10 and determines whether or not to perform image recording. In the present embodiment, image acquisition during continuous shooting is switched between recording image processing, imaging/display processing, and focus detection processing. The switching may be alternated, or, for example, imaging/display and focus detection may be performed once every three times. As a result, highly accurate focus detection can be performed without significantly reducing the number of shots per unit time.

If it is determined in S10 that image recording is to be performed, the process advances to S300 to execute a photographing subroutine. Details of the shooting subroutine will be described later. When the photographing subroutine is executed in S300, the process returns to S9 and it is determined whether or not Sw2 is detected to be on, that is, whether or not continuous photographing is instructed.

If it is determined in S10 that imaging/display and focus detection are to be performed, the process advances to S400 to execute imaging/display and focus detection processing during continuous shooting. The imaging/display and focus detection processing during continuous shooting are the same as those in S8. The difference is that the display period, display update rate (interval), and display delay of the image captured in S400 are different from those in the process of S8, depending on the shooting frame speed of continuous shooting, the generation process of the recorded image, and the like. is. The system control unit 50 as display control means performs the above-described display control. As in the present embodiment, when the display period, update rate, and display delay of the displayed image change during continuous shooting, the photographer's line of sight position is affected to some extent. In the present invention, the line-of-sight position is appropriately processed and the detection process is controlled in view of the fact that an error occurs in the detected line-of-sight position depending on the state and switching of the display specifications described above. As a result, it is possible to obtain a highly accurate line-of-sight position regardless of changes in display specifications. The obtained line-of-sight position information is used, as described above, for setting the focus detection area, linking it with the detected subject area, and the like. Details will be described later. When image pickup/display and focus detection processing during continuous shooting are executed in S400, the process returns to S9 to detect whether Sw2 is ON, that is, whether or not continuous shooting is instructed.

When Sw1 is not detected to be on (or is detected to be off) in S5 and the main switch is detected to be off in S11, focus detection and photographing operations are terminated. On the other hand, if it is not detected in S11 that the main switch is off, the process returns to S2 to acquire image data and focus detection data.

[Explanation of line-of-sight prediction]
Next, the line-of-sight position processing control process for predictive control using the detected line-of-sight position information will be described with reference to FIG. FIG. 6 is a flowchart for explaining the line-of-sight prediction method. The processing in FIG. 6 is executed in parallel by the system control unit 50 and the line-of-sight input operation unit 701 as main subjects after S4 in FIG.

In step S201, the detected line-of-sight position information is acquired within a predetermined period.

The line-of-sight reliability may be calculated by the following method or the like. As a method of obtaining the reliability of the line-of-sight detection data, a method of calculating the variance of the line-of-sight detection positions over a certain time span in the past and taking the reciprocal thereof is considered as a reliability evaluation value of the line-of-sight information. By taking the reciprocal of the distributed data, when the distributed data is small, the line-of-sight information has little variation and the value is stable (high reliability), so the reliability value increases. Conversely, when the distributed data is large, the line-of-sight information has large variations and the value is unstable (reliability is low), so the reliability value is small.

In addition, the longer the focal length, the more the subject being photographed becomes blurred due to the user's camera shake. may be added to calculate the reliability. Specifically, the shorter the focal length, the higher the reliability, and the longer the focal length, the lower the reliability.

In addition to the above, information acquired from the line-of-sight detection sensor itself may be added to the reliability of the line-of-sight information according to the degree of opening of the eyelids. The reason that the reliability of line-of-sight information changes according to the degree of eyelid opening is similar to the reason that the line-of-sight detection accuracy differs according to the line-of-sight position, and is caused by part of the pupil being hidden by the eyelid. A change in the reliability of line-of-sight information according to the degree of eyelid opening can be obtained from a line-of-sight detection sensor. If it is not possible to obtain the reliability of line-of-sight information based on the degree of opening of the eyelids with the line-of-sight detection sensor, information on the degree of opening of the eyelids may be obtained from a separate sensor and the reliability may be evaluated.

In steps S202 to S205, line-of-sight prediction is performed using control parameters.

In the present invention using the Kalman filter, it is desirable to set the number of regression data n to a predetermined fixed value. A method of determining the number of regression data n using the reliability of line-of-sight information is conceivable. When the reliability is high, n is decreased in order to give importance to responsiveness, and when the reliability is low, n is increased in order to emphasize suppression of vibration components. However, it may be difficult to determine an appropriate value for n depending on reliability. Variation in line-of-sight detection varies depending on the photographer and shooting conditions. For example, for a photographer with wide eyes, a large n is always set, and a large delay always occurs, which may impair the comfort of the photographer. Therefore, it is desirable to automatically determine the line-of-sight position with a fixed n instead of determining the optimum n for each photographer by calibration in advance and then dynamically determining n during shooting. . However, since setting n to a fixed value may reduce the line-of-sight responsiveness, it is effective to use a Kalman filter to improve the responsiveness.

[Description of Kalman filter]
The Kalman filter is known to be the optimum filter when assuming the normality of errors in the system and a linear state transition model. Filtering can be performed stably with little delay against errors.

The Kalman filter consists of two types of equations, a state equation and an observation equation, and related processing. In this embodiment, the state equation is a state transition model, that is, an equation that models the line-of-sight position movement, and the observation equation is an equation that describes the line-of-sight position detection system. A detailed example will be described later. Also, the Kalman filter has a prediction step and a filtering step. In the prediction step, a pre-estimated value, which is an estimated value at the current time, is calculated from the value at the previous time according to a state equation (model) given in advance. In the filtering step, a post-estimate value corrected by interpolation is calculated from the observed value at the current time and the pre-estimated value. The interpolation weight is called the Kalman gain and is calculated from the variance of the error of the observed value with respect to the prior estimate. When the variance of the observed value error is small, the posterior estimate is close to the observed value, assuming that the reliability of the observation is high. Conversely, when the variance is large, the posterior estimate is close to the prior estimate. By alternately and repeatedly calculating the prediction step and the filtering step in this manner, the state can be automatically predicted. The error variance described above is also automatically updated by the Kalman filter. Even if the Kalman filter is used, a certain amount of delay cannot be avoided, but if the variance of the error is small, a value close to the observed value at the current time can be obtained as the posterior estimate, so the delay can be minimized. can. However, as described above, n must be set to an appropriate value in advance according to the photographer and shooting conditions.

[Description of the "overfitting (overlearning) problem", which is a problem when using a higher-order polynomial of third or higher order as the state equation of the Kalman filter]
The state equation of the Kalman filter should use an equation that models the state change of the system to be predicted. Predicting the line-of-sight position requires a formula that models the movement of the line of sight. There is no clear model formula. However, when the number of frames captured by a camera is 30 frames per second, the human line of sight often moves continuously for a time length of about 10 frames (about 0.3 seconds). Therefore, in many cases, the line-of-sight position at a certain time during shooting can be accurately predicted from the line-of-sight data for about 10 frames in the past from that time, assuming the continuity of the positions. Based on this concept, a regression equation using the history of line-of-sight data as regression data can be approximately used as a state equation for line-of-sight prediction. Regarding the selection of the regression equation, first, considering the processing load on the imaging device, a linear Kalman filter that can be processed at a low calculation cost is desirable. Therefore, it is desirable that the regression equation used as the state equation of the Kalman filter be a polynomial regression equation. Next, considering the stability of line-of-sight prediction, a low-order polynomial is desirable. The human line of sight moves unconsciously, so in order to display the position of the line of sight on the display device of the camera, and to comfortably select subjects and AF points while looking at it, it is necessary to smooth the line of sight using a filter. Is required. Using a higher-order polynomial regression equation can reduce the error for each point in the regression data, but causes oscillations in the predicted values due to so-called overfitting. Therefore, it is desirable to use a low-order regression formula of 0 to 2 as the regression formula used for predicting the line of sight. A cubic or higher regression equation has one or more points of inflection and has a maximum value and a minimum value, which may cause oscillation.

[Embodiment using a low-order polynomial regression equation as the state equation of the Kalman filter]
In this embodiment, a case where the line-of-sight position is predicted using linear regression and a Kalman filter from past history data of the line-of-sight position will be described with reference to FIGS. 6 and 7. FIG. FIG. 7 shows an example where n=6.

After the process starts, the line-of-sight position is acquired in step S201. At the same time, history data of the past n points of sight lines are accumulated. If the number of accumulated data is m, and m is less than n, the processing of steps S202 to S205 described below may be performed with n=m, or after accumulating line-of-sight data until m=n Processing may begin.

In the flowchart of FIG. 6, step S202 follows immediately after step S201 as the order of processing, but since steps S201 to S205 are repeatedly processed, for the convenience of explanation, step S203 will be described.

After the process is started, the initial value of the pre-estimated value _xq of the line-of-sight position x in step S202 may be the observed value as it is or the value of the coordinate origin of the position. Also, as is well known, a small value greater than 0, such as about 10 ⁻² , may be given to the initial value of the variance P _q of the error e _q of the pre-estimated value, which will be described later.

Let the line-of-sight coordinates be (x, y). For the sake of explanation, the one-dimensional line-of-sight coordinate x will be described here, but the same applies to y.

Assuming that a plurality of frames are continuously shot by live view shooting of an imaging device, the time at the k-th shot frame (hereinafter referred to as frame k) is t(k), and the linear regression equation is expressed as Equation (1). do. t is time and is an independent variable.

FIG. 7(a) is a diagram for explaining linear regression calculation from n pieces of data from time t(k−n+1) to t(k). The calculated regression line is L _f (k).

In step S203, regression coefficients a(k) and b(k) are calculated.

Regression coefficients a(k) and b(k) of equation (1) are calculated from n pieces of regression data by the least-squares method using equations (2) and (3). You may calculate a(k) and b(k) by well-known iterative least-squares method.

From this, the a priori estimated value x _q (k+1) in the next frame, that is, the frame k+1 is the posterior estimated value at times t(k−n+1) to t(k) as regression data, as shown in FIG. _xp is used to calculate in step S204 by Equation (4).

In equation (4), from equations (2) and (3), a(k) and b(k) contain at most first-order terms in x(k), so this is linear with respect to position x(k) The formula is

In this embodiment, the state equation of the Kalman filter is given by Equation (5).

Equation (5) is a linear state equation including regression coefficients in part of A(k) and u(k). Equation (5) is obtained by rearranging x(k) in Equation (4). It is assumed that v(k) is a modeling error component due to slight eye movement, etc., and is a random variable that follows a normal distribution with mean 0 and variance Q(k).

Q(k) may be a known average value of human fixational eye movement, or may be the gaze position measured with respect to an index point whose correct coordinate position is known after calibration of the gaze detection device. For example, a variance may be given.

Let the observation equation of the Kalman filter be Formula (6).

Let w(k) be the measurement error, which is a random variable following a normal distribution with mean 0 and variance R(k).

R(k) may be the error of the line-of-sight detection device that is measured in advance using an index image for which the correct coordinate position is known. It may be dynamically updated every frame or every few frames.

Also, the error e _q (k) of the pre-estimated value and the error q _p (k) of the post-estimated value are defined as in Equations (7) and (8). x _p (k) is the posterior estimate of the gaze position x.

With the variance of e _q (k) as P _q (k) and the variance of e _p (k) as P _p (k), the error variance is also updated by Equation (9).

Next, as shown in FIG. 7C, the posterior estimated value x(k+1) is calculated in the filtering step of the Kalman filter.

First, the Kalman gain G(k) is calculated by Equation (10).

Further, the Kalman gain is used to update the posterior estimated value by Equation (11).

This is a formula for interpolating the posterior estimated value from the prior estimated value and the observed value using the Kalman gain as a weight. When the variance R of the observation error is larger than the error _Pq of the pre-estimated value, the Kalman gain G≈0, and the posterior estimate x _p is close to the pre-estimated value. Conversely, when the error Pq of the pre-estimated value is larger than the variance R of the observed error, the Kalman gain _G≈1 , and the post-estimated value _xp is close to the observed value xm.

The error variance is updated by Equation (12).

Next, as shown in FIG. 7D, the time is advanced by one, and the same processing is repeated in steps S201 to S205.

By repeatedly repeating the above process for each captured frame, it is possible to automatically calculate and predict the line-of-sight position that minimizes the variance of the posterior estimated value according to the size of the error in the observed value. .

In addition, in the description of the embodiment, the case of predicting the line of sight using the continuously acquired line of sight information was described, but there are cases where the line of sight information cannot be partially acquired due to reasons such as blinking or shuttering during the line of sight acquisition. . Even in such a case, the line-of-sight information may be appropriately associated with the timing when the line-of-sight information was acquired, taking into consideration the line-of-sight information for the time when the line-of-sight information could not be obtained. It's not unpredictable.

Further, in the embodiment, an example is shown in which the order of the regression equation is set to the first order, and fixed values are set at the time of photographing. For example, the order of the regression equation may be dynamically changed in consideration of the rate at which detection information can be acquired over a certain time span, the line-of-sight position error over a certain time span, the line-of-sight reliability, and the like.

Also, in the embodiments, an example in which the present invention is implemented in a digital camera has been described, but the present invention may be applied to any device as long as it performs line-of-sight detection. For example, it is also possible to implement in a head mounted display, a smart phone, a PC, or the like.

Also, in the operations described using the flowcharts in the above embodiments, it is possible to change the order of the steps to be executed as appropriate so as to achieve the same purpose.

The present invention can also be configured to supply a program implementing one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium. . It can also be realized by processing in which one or more processors in the computer of the system or apparatus read and execute the program. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Claims (4)

acquisition means for acquiring line-of-sight information corresponding to the line-of-sight position of a person;
and a correction means for correcting the line-of-sight information,
The processing device, wherein the correction means corrects the line-of-sight information by a Kalman filter using a low-order polynomial regression equation. 3. A processing apparatus according to claim 2, wherein said correction means uses a polynomial of degree 0 to 2 as the low-order polynomial. having display means for displaying an image,
4. The processing apparatus according to any one of claims 1 to 3, wherein the acquisition unit acquires, as the line-of-sight information, information corresponding to a line-of-sight position on the display unit at which a person gazes. . an acquisition step of acquiring line-of-sight information corresponding to the line-of-sight position of a person;
a correction step of correcting the line-of-sight information;
The method of controlling a processing device, wherein in the correcting step, the line-of-sight information is corrected using a Kalman filter.

Images