JP2017121042A - Subject tracking apparatus, control method of the same, control program, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent malfunction of tracking caused by erroneous tracking of a subject.SOLUTION: A camera 101 is configured to track a subject in continuously obtained photographic images while shifting the subject. In the photographic image, a CPU 105 obtains a position of the subject in the photographic image and reliability that represents subject probability. Then, the CPU 105 controls a correction lens 114 to move the position of the subject to a target position in the photographic image and changes a tracking state of the subject on the basis of the reliability.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a subject tracking device, a control method thereof, a control program, and an imaging device.

  In general, in an imaging apparatus such as a digital camera, important photographing processes such as exposure determination and focusing are automated. Furthermore, if an image pickup apparatus equipped with an image stabilization control device for preventing image blur due to camera shake or the like is used, it is possible to substantially eliminate a factor that causes a user's shooting mistake at the time of shooting.

  On the other hand, when shooting a moving subject or shooting at telephoto where the focal length is large, there are the following problems.

  When shooting a moving subject, if the subject moves off the shooting screen, a special technique of the user is required to accurately track the subject by a user operation. In addition, when shooting with an imaging apparatus having a telephoto lens, the influence of image blur due to camera shake becomes large. This makes it difficult for the user to maintain the subject at the center of the shooting screen. Even if the user operates the imaging device to return the subject to the shooting screen, the amount of camera shake caused by the operation is corrected by the imaging device. As a result, it becomes difficult to finely adjust the subject to the shooting screen or the center of the shooting screen due to the influence of the vibration control.

  In order to cope with such a problem, for example, there is an imaging device including a tracking control device that automatically tracks a subject by moving a part of the optical system in a direction intersecting the optical axis (see Patent Document 1). . Further, an imaging apparatus that extracts a target subject from an image obtained as a result of shooting and tracks the subject with a rotary head so that the center of gravity of the subject is located near the center of the shooting screen. Yes (see Patent Document 2).

JP 2010-93362 A JP-A-7-226873

  However, in the imaging apparatus described in Patent Document 2, the estimation accuracy of the centroid position extracted from the captured image affects the tracking control of the rotary head. In the following description, extraction of a specific subject area from sequentially supplied captured images is called tracking, and a part of the optical system or a movable part such as a rotary head is controlled based on the subject position based on the tracking result. This is called tracking.

  In tracking an object, if an area similar to the tracking target exists in the captured image, an incorrect area may be tracked. If tracking is performed based on an incorrect tracking result, shooting by the user is greatly hindered.

  Accordingly, an object of the present invention is to provide a subject tracking device, a control method thereof, a control program, and an imaging device that can prevent a tracking malfunction caused by a tracking error of the subject.

  In order to achieve the above object, a subject tracking device according to the present invention includes a shift unit that shifts a subject in a photographed image, and a subject that obtains reliability indicating the position of the subject and the likelihood of the subject in the photographed image. Tracking means; and control means for controlling the shift means to move the position of the subject in the captured image to a target position in the captured image and changing the tracking state of the subject based on the reliability. It is characterized by having.

  An imaging apparatus according to the present invention includes an imaging unit that obtains a captured image according to an optical image formed through an imaging optical system, and the subject tracking apparatus.

  A control method according to the present invention is a control method of a subject tracking device that includes a shift unit that shifts a subject in continuously obtained captured images, and that tracks the subject, in the captured image, the subject in the captured image A subject tracking step for obtaining a reliability indicating the position and the likelihood of the subject, and controlling the shift means to move the position of the subject in the captured image to a target position in the captured image, and based on the reliability And a control step for changing the tracking state of the subject.

  A control program according to the present invention is a control program used in a subject tracking device that includes a shift unit that shifts a subject in continuously obtained captured images and that tracks the subject, and is a computer provided in the subject tracking device. In addition, in the captured image, a subject tracking step for obtaining a reliability indicating the position of the subject and the likelihood of the subject in the captured image, and the position of the subject in the captured image by controlling the shift means. And a control step of changing the tracking state of the subject based on the reliability.

  According to the present invention, it is possible to prevent a tracking malfunction caused by an object tracking error.

It is a figure which shows typically the external appearance about an example of an imaging device provided with the subject tracking apparatus by the 1st Embodiment of this invention. It is a figure which shows the example about the structure of the camera shown in FIG. FIGS. 3A and 3B are diagrams for explaining tracking of a subject detected by the camera shown in FIG. 2, in which FIG. 3A is a diagram illustrating a captured image before the tracking of the subject is started, and FIG. FIG. 3 is a flowchart for explaining an example of a subject tracking process performed by a subject tracking unit shown in FIG. 2. It is a figure for demonstrating the template matching process performed in the subject tracking part shown in FIG. It is a block diagram for demonstrating an example of the tracking amount calculation part shown in FIG. 3 is a flowchart for explaining an example of subject tracking performed by the camera shown in FIG. 2. It is a figure which shows the example about the structure of the camera which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating the notification method in the camera shown in FIG.

  BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram illustrating an example of a subject tracking device, a control method thereof, a control program, and an imaging device including the subject tracking device that can reduce a tracking malfunction caused by a subject tracking error according to an embodiment of the present invention. Will be described with reference to FIG. In both of the following first and second embodiments, an example in which the present invention is applied to a digital camera as an imaging device will be described. However, the present invention is not limited to a digital camera, and a digital video camera, surveillance The present invention can also be applied to an imaging apparatus such as a camera, a web camera, or a mobile phone. In addition, the present invention can be applied to either a lens interchangeable type camera or a lens integrated type camera.

[First Embodiment]
FIG. 1 is a diagram schematically illustrating an appearance of an example of an imaging apparatus including a subject tracking device according to the first embodiment of the present invention.

  The illustrated imaging apparatus is, for example, a digital camera (hereinafter simply referred to as a camera) 101, and a release button 104 is disposed on the upper surface of the camera housing. Here, an axis extending in the direction of the side surface of the camera 101 is defined as an X axis, and an axis extending in the direction of the upper surface is defined as a Y axis. An axis extending in the front direction of the camera 101 is taken as a Z axis. Then, the rotation 103Pp around the X axis is set as a pitch (PITCH), and the rotation 103y around the Y axis is set as a yaw (YAW).

  FIG. 2 is a diagram showing an example of the configuration of the camera shown in FIG.

  1 and 2, a correction lens 114 and an image sensor 106 are positioned on the optical axis 102 of the image pickup optical system. The image sensor 106 generates an electrical signal (analog signal) corresponding to the optical image formed through the imaging optical system, A / D converts the analog signal, and outputs it as a captured image. Although not shown, the imaging optical system includes a zoom lens and a focus lens. The angular velocity meter 103 detects an angular shake (that is, camera shake) at the pitch 103p and the yaw 103y and sends an angular shake signal to the CPU 105. The CPU 105 constitutes the subject tracking device of the present embodiment.

  In the CPU 105, a shake correction angle calculation unit 108 obtains a shake correction angle based on the angle shake signal. For example, the shake correction angle calculation unit 108 cuts a DC component added as detection noise in the angle shake signal, performs integration processing, and outputs an angle signal indicating the angle of the camera 101. For example, an HPF (High Pass Filter) is used for cutting the DC component. Then, the angle signal is sent to the sensitivity adjustment unit 109.

  A zoom position and a focus position indicating the positions of the zoom lens and the focus lens are sent from the zoom and focus position detection unit 107 to the sensitivity adjustment unit 109. The sensitivity adjustment unit 109 obtains the focal length and the shooting magnification based on the zoom position and the focus position. Then, the sensitivity adjustment unit 109 amplifies the angle signal according to the focal length and the shooting magnification to obtain a shake correction target value. The sensitivity adjustment unit 109 sends the shake correction target value to the drive control unit 113 as a shake correction amount.

  Note that the shake correction sensitivity on the camera image plane with respect to the shake correction stroke of the correction lens 114 changes due to the change in the position of the focus lens and the zoom lens. Therefore, here, the shake correction target value is obtained based on the zoom position and the focus position. .

  The correction lens 114 described above is used to shift and move a tracking target subject (tracking target subject) in an image obtained as a result of shooting. As will be described later, the drive control unit 113 drives the correction lens 114 to track the subject. Further, the drive control unit 113 corrects image blur by driving the correction lens 114 in a direction different from the optical axis (for example, a crossing direction) (optical image stabilization).

  In the example shown in FIG. 2, optical image stabilization is performed using the correction lens 114. However, as an image blur correction method, the image blur correction method is performed by moving the image sensor 106 in a plane orthogonal to the optical axis. May be used. Furthermore, electronic image stabilization that reduces the influence of shake by changing the cutout position of the image for each shooting frame that is the output of the image sensor 106 may be used. A plurality of image blur correction methods described above may be used in combination.

  The subject tracking unit 110 receives the captured image that is the output of the image sensor 106 and extracts the position of the subject region in the captured image. In the present embodiment, the position of the center of gravity of the subject area is used as the subject position. Furthermore, the subject tracking unit 110 obtains a reliability (likelihood) indicating the likelihood of the subject, and sends the center of gravity position and the likelihood to the tracking amount calculation unit 111. The tracking amount calculation unit 111 calculates a tracking correction amount that is a control amount used for tracking the subject by the correction lens 114 based on the position of the center of gravity of the subject region. Further, the tracking amount calculation unit 111 changes the tracking correction amount based on the likelihood.

  The adder 112 adds the shake correction amount that is the output of the sensitivity adjustment unit 109 and the tracking correction amount that is the output of the tracking amount calculation unit 111, and sends the addition result to the drive control unit 113. The drive control unit 113 calculates the drive amount of the correction lens 114 based on the addition result that is the output of the adder 112, drives the correction lens 114 based on the drive amount, and performs subject tracking and image blur correction. Do.

  FIG. 3 is a diagram for explaining tracking of a subject detected by the camera shown in FIG. FIG. 3A is a diagram illustrating a captured image before the subject tracking is started, and FIG. 3B is a diagram illustrating a captured image after the subject tracking is started.

  In the captured image 301 a shown in FIG. 3A, the subject 302 a exists at a position away from the image center 304. Here, the gravity center position (subject gravity center position) of the subject 302a is indicated by reference numeral 303a. The CPU 105 gradually brings the subject gravity center position 303a closer to the image center 304 by tracking (also referred to as tracking control), and finally makes the image center 304 and the subject gravity center position 303a substantially coincide with each other. As a result, as shown in FIG. 3B, the subject gravity center position 303a of the subject 302a that has been successfully tracked coincides with the image center 304.

  Here, a subject tracking method in the subject tracking unit 110 will be described. In the illustrated example, when tracking an object, a method of estimating a region with a low degree of difference by using a partial image indicating a target object as a template and matching the captured image while shifting the template (template matching). Is used. Further, in order to cope with the scale conversion in the time direction for the target subject, a method of extracting the subject feature color and estimating the subject region based on the distribution state of the feature color of the photographed image is also used. Then, the template is updated based on the estimated subject area.

  FIG. 4 is a flowchart for explaining an example of subject tracking processing performed by the subject tracking unit 110 shown in FIG.

  When the subject tracking process is started, the subject tracking unit 110 reads a captured image from the image sensor 106 (step S401). Then, the subject tracking unit 110 determines whether there is a subject that is a tracking target designated in advance in the captured image (step S402). If the tracking target subject does not exist (NO in step S402), the subject tracking unit 110 detects the subject in the captured image in order to determine the tracking target subject (step S403).

  In the process of step S403, the subject tracking unit 110 detects a subject based on a user instruction or automatically detects a subject. When a subject is detected based on a user's instruction, the user instructs the position of the subject in the captured image using an input interface having a touch panel and buttons. The subject tracking unit 110 detects the subject based on the position designated by the user and extracts the subject region.

  On the other hand, when automatically detecting a subject, the subject tracking unit 110 uses face detection. When detecting a face, for example, a technique that uses knowledge about the face (skin color information, parts such as eyes, nose, mouth, etc.) and a technique that constitutes a classifier for face detection by a learning algorithm represented by a neural network, etc. There is. In face detection, face detection is generally performed by combining the above methods in order to improve the detection rate. As for face detection, for example, a method of performing face detection using a wavelet transform and an image feature amount described in JP-A-2002-251380 is known.

  Subsequently, the subject tracking unit 110 extracts the feature amount of the subject that is the tracking target from the subject region (step S404). Here, since the tracking process is performed using template matching, the image pattern of the subject area is extracted as the feature amount. Furthermore, since the subject area is estimated based on the distribution of characteristic colors, the color histogram Hin of the subject area is held. Thereafter, the subject tracking unit 110 ends the subject tracking process and waits until the next imaging sampling period.

  On the other hand, if there is a subject to be tracked (YES in step S402), subject tracking unit 110 performs a template matching process using a template (step S405).

  FIG. 5 is a diagram for explaining template matching processing performed by the subject tracking unit 110 shown in FIG. FIG. 5A is a diagram showing an example of a subject model (template) in the template matching process, and FIG. 5B is a diagram showing an example of an image for searching for a subject to be tracked.

  Here, the pixel pattern of the partial image (template) 501 showing the subject to be tracked is used as the feature amount. The luminance of the pixel data is used as the feature quantity 502 of the template 501. The feature amount T (i, j) is expressed by the following equation (1), where the coordinates in the template are (i, j), the number of horizontal pixels is W, and the number of vertical pixels is H.

  In FIG. 5B, the coordinates in the search image 503, which is the range in which matching processing is performed, are assumed to be (x, y). A partial area 504 for obtaining an evaluation value of template matching in the search image 503 is set. The brightness of the captured image is used as the feature quantity 505 of the partial area 504 in the same manner as the template 501. The feature quantity S (i, j) is expressed by the following equation (2), where the coordinates in the partial region 504 are (i, j), the number of horizontal pixels is W, and the number of vertical pixels is H.

  When evaluating the similarity between the template 501 and the partial region 504, an absolute difference sum, a so-called SAD (Sum of Absolute Difference) value, is used. And a SAD value is calculated | required by following Formula (3).

  The SAD value V (x, y) is obtained while shifting the partial area 504 one pixel at a time from the upper left of the search image 503. A coordinate (x, y) at which the SAD value V (x, y) indicates a minimum value indicates a position most similar to the template 501. That is, the position indicating the minimum value is a position where there is a high possibility that the target subject to be tracked exists in the search image 503.

  Here, an example in which one-dimensional information such as luminance (luminance signal) is used as the feature amount has been described, but three-dimensional information such as lightness, hue, and saturation may be used as the feature amount. Further, although the SAD value is used when obtaining the template matching evaluation value, a method such as normalized cross-correlation, so-called NCC (Normalized Correlation Coefficient) may be used.

  Referring to FIG. 4 again, the subject tracking unit 110 estimates the subject region based on the position estimated by the template matching process (step S406). Here, the subject tracking unit 110 uses the color histogram Hin of the subject area obtained in step S404 and the color histogram Hout of the entire or part of the captured image at the current time to obtain information represented by the following equation (4). The quantity I (a) is determined.

  The information amount I (a) indicates the occurrence probability of the subject area with respect to the entire captured image or a part thereof in each bin of the color histogram. The subject tracking unit 110 adapts the information amount I (a) to each pixel of the captured image at the current time to generate a map indicating the possibility that the subject exists in the search image 503. Then, the subject tracking unit 110 estimates the subject area based on the map, and outputs the estimated gravity center position of the subject area.

  Subsequently, the subject tracking unit 110 calculates the likelihood of subject tracking (step S407). Factors that hinder the likelihood of subject tracking include, for example, subject changes, the presence of similar subjects, and accumulation of tracking errors. The subject tracking unit 110 calculates the likelihood by multiplying these factors by the evaluation value obtained from the template matching process and the estimation of the subject region.

  The larger the minimum value of the SAD value V (x, y) obtained by the above equation (3), the greater the change in the subject. Therefore, the subject tracking unit 110 decreases the likelihood as the minimum value increases. If the SAD value similar to the minimum value of the SAD value V (x, y) obtained by Expression (3) is present at a position that is separated from the estimated position of the subject by a predetermined threshold or more, a similar subject exists. Probability is high. For this reason, the subject tracking unit 110 determines the similarity between the SAD value present at a position away from the estimated position of the subject by a predetermined threshold or more and the minimum value of the SAD value V (x, y) obtained by Expression (3). The higher the value, the lower the likelihood.

  Regarding the information amount I (a) indicating the color characteristic color of the subject obtained by Expression (4), the change in the subject increases as the entropy Ein, which is an average value (expected value) in the subject region, decreases. Entropy Ein is expressed by the following equation (5).

  The subject tracking unit 110 decreases the likelihood as the entropy Ein decreases. Furthermore, with respect to the information amount I (a) that is the characteristic color of the subject represented by Expression (4), the possibility that a similar target exists increases as the entropy Eout, which is an average value (expected value) outside the subject area, increases. Entropy Eout is expressed by the following equation (6).

  The subject tracking unit 110 decreases the likelihood as the entropy Eout increases. Furthermore, once the subject tracking accuracy is lowered, the reliability of the subsequent tracking is also lowered. For this reason, the subject tracking unit 110 considers the likelihood of the history when calculating the likelihood. For example, an average value of likelihoods for a predetermined period is used as the likelihood of the current frame. In this way, the subject tracking unit 110 calculates the likelihood of subject tracking.

  Subsequently, the subject tracking unit 110 updates the subject feature amount (step S408). Here, the subject tracking unit 110 updates the template based on the subject area estimated in the process of step S406, and responds to the scale change of the subject. Thereafter, the subject tracking unit 110 ends the subject tracking process and waits until the next imaging sampling period.

  FIG. 6 is a block diagram for explaining an example of the tracking amount calculation unit 111 shown in FIG.

  The tracking amount calculation unit 111 calculates the tracking correction amount in each of the vertical direction and the horizontal direction of the image. Here, calculation of the tracking correction amount in one direction will be described. The tracking amount calculation unit 111 calculates a count value for tracking the subject based on the difference between the position of the subject (that is, the gravity center position) and the image center position (target position of the subject). The tracking amount calculation unit 111 controls the position of the subject to move to the target position by adding the count value every sampling time. Then, the tracking amount calculation unit 111 changes the degree of tracking by changing the size of the count value based on the likelihood (subject likelihood) output from the subject tracking unit 110.

  The subtracter 604 subtracts the coordinates of the image center position acquired by the image center position acquisition unit 602 from the coordinates of the subject position acquired by the subject position acquisition unit 601. Thereby, a distance (center deviation amount) on the image between the image center position and the gravity center position of the subject is calculated. This center shift amount is signed data when the image center is zero. The output of the subtractor 604 is input to the count value table 605, and a count value corresponding to the difference distance between the subject gravity center position and the image center is output from the count value table 605. Note that the subject position acquisition unit 601 may acquire a subject position on a coordinate centered on the image center position (both x and y coordinates are zero).

  The tracking amount calculation unit 111 sets the count value to zero when the center deviation amount is equal to or smaller than the threshold value Z or equal to or larger than the threshold value −Z. As a result, a dead zone area in which tracking is not performed within the range of ± Z from the image center is set. The count value table 605 is a table in which the count value increases as the center deviation amount increases, and the sign of the count value is matched with the sign of the center deviation amount.

  The output of the count value table 605 is input to the variable gain device 606. The gain amount calculation unit 607 calculates a gain (control gain) related to the count value based on the subject likelihood acquired by the subject likelihood acquisition unit 603. Here, when the subject likelihood is high, the gain amount calculation unit 607 increases the gain in order to increase the tracking responsiveness on the assumption that the correct subject is likely to be tracked. On the other hand, when the subject likelihood is low, there is a possibility that an erroneous subject has been tracked. If tracking is performed on the subject that is the wrong tracking target, the user's shooting is greatly hindered. For this reason, the gain amount calculation unit 607 reduces the gain when the subject likelihood is low. At this time, when the subject likelihood is equal to or less than the threshold, the tracking control may be stopped by setting the gain to 0. Further, the tracking control may be stopped by turning off the tracking switch 608 instead of setting the gain to zero. Then, the gain amount calculation unit 607 sets the calculated gain (gain amount) in the variable gain unit 606 as the variable gain Cg.

  The output of the variable gain unit 606 is input to the signal selection unit 609. The signal selection unit 609 receives the down count value of the down count value setting unit 611 and the setting of the tracking switch 608. The signal selection unit 609 selects the output of the variable gain unit 606 when the tracking switch 608 is on. On the other hand, when the tracking switch 608 is off, the signal selection unit 609 selects the output of the downcount value setting unit 611. The output of the signal selection unit 609 is input to the adder 610.

  The down count value setting unit 611 sets a down count value. The tracking amount previous sampling value is input from the sampling unit 613 to the downcount value setting unit 611. The down count value setting unit 611 sets the down count value to minus when the tracking amount previous sampling value is a plus sign. On the other hand, when the tracking amount previous sampling value is a minus sign, the downcount value setting unit 611 sets the downcount value to a plus. Thereby, the downcount value setting unit 611 decreases the absolute value of the tracking correction amount. The downcount value setting unit 611 sets the downcount value to zero when the tracking amount previous sampling value is within 0 ± predetermined range. The tracking amount previous sampling value indicates a tracking correction amount up to the previous sampling.

  The adder 610 adds the output of the signal selection unit 609 and the tracking amount previous sampling value. When a negative down count value is added to the tracking amount previous sampling value, the absolute value of the tracking correction amount becomes small. The output of the adder 610 is input to the upper / lower limit value setting unit 612. The upper / lower limit value setting unit 612 sets the tracking correction amount to an amount that is less than the predetermined upper limit value and exceeds the predetermined lower limit value. The output of the upper / lower limit setting unit 612 is input to the sampling unit 613 and the LPF 614. The LPF 614 sends the tracking correction amount from which the high frequency noise has been cut to the correction lens amount conversion unit 615. The correction lens amount conversion unit 615 converts the tracking correction amount into a signal form for tracking by the correction lens 114, and outputs the final tracking correction amount.

  In this way, the tracking amount calculation unit 111 obtains a count value for each sampling according to the difference between the image center position and the subject position, adds it to the tracking correction amount, and gradually brings the subject position closer to the vicinity of the image center. Perform tracking.

  In the above description, the gain amount calculation unit 607 sets the gain based on the subject likelihood, but the gain amount calculation unit 607 is based on the center of gravity position of the captured image in addition to the subject likelihood. A gain may be set.

  As described above, the subject tracking unit 110 estimates a target subject area from the captured image. For this reason, it is necessary to perform tracking so that the target subject is contained in the captured image. When the target subject exists at the end of the captured image, there is a high possibility that the subject will deviate from the captured image after the next frame. Therefore, it is important to move the subject to the center of the image (target position) by tracking.

  On the other hand, when the target subject exists in the vicinity of the center of the image (target position), it is unlikely that the subject will deviate from the captured image after the next frame. Therefore, the importance of tracking decreases. Therefore, the lower the subject likelihood, the smaller the tracking degree (that is, the tracking state) is controlled, and the closer the subject gravity center position is to the image center (target position), the smaller the tracking degree is controlled.

  FIG. 7 is a flowchart for explaining an example of subject tracking performed by the camera shown in FIG. The process according to the flowchart shown in the figure starts when the main power source of the camera 101 is turned on and is performed at a predetermined sampling period.

  First, the CPU 105 determines whether or not an anti-shake SW (not shown) is ON (step S701). If the image stabilization SW is ON (YES in step S701), the CPU 105 captures the output of the angular velocity meter 103 (step S702). Then, the CPU 105 determines whether or not the camera 101 is in a state where shake correction is possible (step S703). Here, the CPU 105 determines that the shake correction cannot be performed when the camera 101 is supplied with power and the output of the angular velocity meter 103 is stabilized. On the other hand, when the camera 101 is in a state after the output of the angular velocity meter 103 is stabilized, the CPU 105 determines that the shake correction is possible. As a result, shake correction is not performed when the output of the angular velocity meter 103 is unstable immediately after the power is supplied.

  When the camera 101 is in a state in which shake correction is possible (YES in step S703), the CPU 105 outputs the angular velocity meter 103 using the shake correction angle calculation unit 108 and the sensitivity adjustment unit 109 as described above. Based on the above, a shake correction amount is obtained (step S704). On the other hand, when the camera 101 is not in a state where shake correction is not possible (NO in step S703), the CPU 105 sets the shake correction amount to zero (step S705). If the image stabilization SW is OFF (NO in step S701), the CPU 105 proceeds to the process in step S705.

  After the process of step S704 or S705, the CPU 105 determines whether or not the tracking SW (not shown) is ON (step S706). If the tracking SW is ON (YES in step S706), the CPU 105 determines whether there is a subject to be tracked in the captured images continuously obtained from the image sensor 106 (step S707). If there is a subject to be tracked (YES in step S707), CPU 105 estimates the position of the center of gravity of the subject region in the captured image (step S708).

  Subsequently, the CPU 105 obtains the subject likelihood related to the center of gravity position as described above (step S709). As described above, the processing of steps S708 and S709 is performed by the subject tracking unit 110. Then, the CPU 105 obtains a tracking correction amount based on the subject gravity center position and the subject likelihood by the tracking amount calculation unit 111 (step S710).

  If there is no subject to be tracked (NO in step S707), the CPU 105 sets the tracking correction amount to zero (step S711). If tracking SW is OFF (NO in step S706), CPU 105 proceeds to the process of step S711.

  After the process of step S710 or S711, the CPU 105 adds the shake correction amount and the tracking correction amount to calculate the lens driving amount (step S712). Then, the CPU 105 causes the drive control unit 113 to drive and control the correction lens 114 based on the lens driving amount (step sS713). Thereafter, the CPU 105 ends the subject tracking process and waits until the next sampling cycle.

  Thus, in the first embodiment, the correction lens 114 is driven and controlled based on the position of the center of gravity of the subject obtained by tracking the subject and its likelihood. As a result, tracking malfunction can be prevented and tracking with high responsiveness can be performed.

[Second Embodiment]
In the first embodiment described above, the imaging apparatus including the subject tracking device that changes the tracking degree based on the subject likelihood has been described. In contrast, in the second embodiment, the tracking degree is changed based on the subject likelihood, and when the tracking control is difficult, the photographer is notified (warned) that the tracking control is difficult. An imaging device including a subject tracking device that instructs the above will be described. Whether tracking control is difficult is determined based on the subject likelihood. Note that the point of changing the tracking degree based on the subject likelihood is the same as in the first embodiment, and a description thereof will be omitted. Further, as in the first embodiment, a mode in which the subject tracking device is applied to the camera will be described.

  FIG. 8 is a diagram showing an example of the configuration of a camera according to the second embodiment of the present invention. In the second embodiment, the CPU 105 as the subject tracking device determines whether the tracking state determination unit 801 and the tracking state determination unit 801 determine whether tracking control is difficult based on the subject likelihood. The second embodiment is different from the first embodiment in that it includes a notification instruction unit 802 for instructing a photographer to be notified. The camera 101 of the second embodiment includes a display unit (FIGS. 9A to 9D) that displays an image and a display control unit 803 that controls the image displayed on the display unit. Upon receiving the notification instruction from the notification instruction unit 802, the display control unit 803 notifies the photographer that tracking control is difficult.

  Similar to the first embodiment, the subject tracking unit 110 acquires subject likelihood for the set subject to be tracked. The acquired subject likelihood is sent to the tracking state determination unit 801. The tracking state determination unit 801 determines whether the subject likelihood is equal to or less than a threshold based on the subject likelihood received from the subject tracking unit 110. When the received subject likelihood is equal to or smaller than the threshold, the tracking state determination unit 801 determines that tracking control is difficult because the subject likelihood is low. On the other hand, when the received subject likelihood is larger than the threshold value, the tracking state determination unit 801 determines that the subject likelihood is higher than the level at which tracking control is possible. The determination result by the tracking state determination unit 801 is sent to the notification instruction unit 802. When the tracking state determination unit 801 determines that the subject likelihood is equal to or less than the threshold value, the notification instruction unit 802 instructs the display control unit 803 to notify the photographer that tracking control is difficult. send. Upon receiving a notification instruction to the photographer from the notification instruction unit 802, the display control unit 803 changes the content of the image displayed on the display unit and notifies the photographer that tracking control is difficult.

  Next, a method for notifying the photographer that tracking control is difficult will be described with reference to FIG. As shown in FIGS. 9A to 9D, the camera 101 includes a liquid crystal display 901 that functions as a display unit on the back surface, and displays an image captured by the camera 101 before shooting by live view display. can do. In addition, a touch panel 902 is disposed below the liquid crystal display 901 in the camera 101, and the coordinate position of the subject 903 designated by the touch operation can be acquired and the subject to be tracked can be set. A subject 903 designated by the touch operation is indicated by a subject frame 904. Upon receiving a notification instruction from the notification instruction unit 802, the display control unit 803 changes the display of the tracking icon 905 displayed on the liquid crystal display 901 to notify the photographer that tracking control is difficult. An example of a change in the tracking icon 905 according to the change in the tracking control state will be described.

  In FIG. 9A, when the subject 903 to be tracked is specified by the touch operation, the tracking control is performed and the screen of the liquid crystal display 901 (hereinafter simply referred to as “tracking control”) is understood. A tracking icon 905 is displayed on the screen. The tracking icon 905 is not displayed on the screen while the subject tracking is not being performed (the subject is not specified).

  FIG. 9B is a diagram showing a state after the subject is designated. As the time elapses, the shake correction amount and the tracking correction amount when the camera state changes in the order of FIGS. 9A, 9B, 9C, and 9D are added. The output of the adder 112 is shown in FIG. FIG. 9F shows the change in the subject position, FIG. 9G shows the change in the subject likelihood, and FIG. 9H shows the display state of the tracking icon 905. The tracking icon 905 is not displayed on the screen until the timing T11 when the subject is specified by the touch operation as shown in FIG. 9A, and the subject to be tracked is shown in FIG. 9B. When designated, a tracking icon 905 is displayed on the screen. After that, even if the designated subject 903 is away from the center of the image, the tracking control is performed based on the tracking correction amount calculated by the tracking amount calculation unit 111, so that the subject 903 is shown in FIG. 9B. Is returned to the center of the image. In the drawing, the timing when the subject is specified and the position is detected is indicated as T11, and the timing when the subject is returned to the center of the image by the tracking control is indicated as T12.

  Thereafter, when the subject 903 further moves as shown in FIG. 9C, the tracking correction amount is further increased and tracking control is performed. When a plurality of similar subjects appear on the screen as shown in FIG. 9D, the subject likelihood is set low. In the present embodiment, when the subject likelihood is equal to or less than the threshold value Th1, the tracking gain is set to zero and the tracking control is not performed. At this time, in order to notify the photographer that no further tracking is performed, the tracking icon 905 is displayed in gray, and the photographer is notified of the tracking impossible state (see FIG. 9H).

  As described above, when the subject tracking device according to the second embodiment determines that the subject likelihood is low and tracking control cannot be performed, the subject tracking device transmits an instruction to change the display of the tracking icon 905. Accordingly, it is possible to prompt the photographer to frame the camera himself and move the subject to the center of the image. Note that the threshold for determining whether or not to perform tracking control and the threshold for notifying the photographer that tracking control is difficult may be the same or different. When the threshold values match, it is possible to notify the photographer that tracking control is not performed. When the threshold for determining whether or not to perform tracking control is smaller, it is possible to notify the photographer that the subject likelihood is low while performing tracking control, and to prompt the photographer to perform framing before tracking control stops Therefore, tracking control can be performed more smoothly than when tracking control is once stopped and then restarted.

  In the second embodiment, the photographer is notified that the tracking control is difficult by changing the color of the tracking icon, but the notification method is not limited to this. In addition to the tracking icon, an icon indicating that the subject likelihood is low may be displayed on the liquid crystal display 901, or an LED lamp is provided in the camera 101, and the lighting state (lighted, turned off, blinking, etc.) of the LED lamp is set as the subject. It may be changed when the likelihood is high (when tracking control is performed).

  In the above-described embodiment, the case where the correction lens is used as the shake correction member and the correction lens is applied to the optical image stabilization in which the correction lens is moved in a plane intersecting (for example, orthogonal to) the optical axis has been described. On the other hand, the present invention is not limited to optical image stabilization, subject tracking for moving the image sensor in a plane orthogonal to the optical axis, subject tracking for changing the cutout position in each of the shooting frames output by the image sensor, and the image sensor. The present invention can also be applied to subject tracking that rotationally drives a lens barrel including a photographic lens group, subject tracking that combines a pan head that pans and tilts another camera, or a combination of the plurality of subject tracking described above.

  As is clear from the above description, in the example shown in FIG. 2, the correction lens 114 and the CPU 105 function as a shift unit, and the CPU 105, the angular velocity meter 103, and the zoom and focus position detection unit 107 function as a subject tracking unit. The CPU 105 functions as a control unit. At least the correction lens 114, the CPU 105, the angular velocity meter 103, and the zoom and focus position detection unit 107 constitute a subject tracking device.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above-described embodiment may be used as a control method, and this control method may be executed by the subject tracking device. Further, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the subject tracking device. The control program is recorded on a computer-readable recording medium, for example.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 103 Angular velocity meter 107 Zoom and focus position detection part 108 Shake correction angle calculation part 109 Sensitivity adjustment part 110 Subject tracking part 111 Tracking amount calculation part 113 Drive control part 114 Correction lens

Claims (11)

A subject tracking device,
Shift means for shifting the subject in the captured image;
In the captured image, subject tracking means for obtaining a reliability indicating the position of the subject and the likelihood of the subject in the captured image;
Control means for controlling the shift means to move the position of the subject in the captured image to a target position in the captured image, and for changing the tracking state of the subject based on the reliability;
A subject tracking device characterized by comprising:   The subject tracking means uses the partial area indicating the subject in the captured image as a template, and collates the template with a partial area of the captured image obtained after the captured image obtained from the partial image used as the template. The subject tracking device according to claim 1, wherein an evaluation value is obtained and the reliability is obtained according to a distribution of the evaluation value.   The subject tracking means estimates a subject area based on a distribution of the characteristic color of the subject in the captured image, and obtains the reliability based on a distribution of characteristic colors inside and outside the subject area. The subject tracking device according to claim 1.   4. The object tracking unit according to claim 1, wherein when the reliability is obtained, the reliability of the subject in the current photographed image is obtained with reference to the reliability history. The subject tracking device according to the item.   The control unit controls the shift unit so as to reduce a difference between the position of the subject and the target position in the captured image, and changes a control gain in the control of the shift unit based on the reliability. The subject tracking device according to claim 1, wherein the tracking state of the subject is changed.   The subject tracking device according to claim 5, wherein the control unit decreases the tracking state of the subject as the reliability is lower.   The subject tracking device according to claim 6, wherein the control unit reduces the tracking state of the subject as the position of the subject in the captured image is closer to the target position. Imaging means for obtaining a captured image according to an optical image formed through an imaging optical system;
A subject tracking device according to any one of claims 1 to 7,
An imaging device comprising:   The imaging apparatus according to claim 8, wherein the shift unit further includes a correction lens that is provided in the imaging optical system and is driven in a direction intersecting an optical axis of the imaging optical system. A control method for a subject tracking device that includes a shift unit that shifts a subject in continuously obtained captured images, and that tracks the subject,
In the captured image, a subject tracking step for obtaining a reliability indicating the position of the subject and the likelihood of the subject in the captured image;
A control step of controlling the shift means to move the position of the subject in the captured image to a target position in the captured image, and changing the tracking state of the subject based on the reliability;
A control method characterized by comprising: A control program that includes a shift unit that shifts a subject in continuously obtained captured images, and that is used in a subject tracking device that tracks the subject,
In the computer provided in the subject tracking device,
In the captured image, a subject tracking step for obtaining a reliability indicating the position of the subject and the likelihood of the subject in the captured image;
A control step of controlling the shift means to move the position of the subject in the captured image to a target position in the captured image, and changing the tracking state of the subject based on the reliability;
A control program characterized by causing

Images