JP6971808B2 - Image shake correction device, image pickup device and control method - Google Patents

Description

The present invention relates to an image shake correction device, an image pickup device, and a control method.

An image pickup device that corrects image shake by detecting shake applied to the image pickup device using a motion vector and driving a shake correction means based on the detected shake information has been proposed. This image pickup device compares the captured image with the image one frame or more before, and uses the movement amount of the representative point (movement amount between frames) as the shake information of the image pickup device. The calculation of the motion vector is accompanied by a time delay of at least one frame or more due to the influence of the image accumulation time and readout time, the image matching time, and the like.

When the image pickup apparatus drives the shake correction means to perform image shake correction based on the shake information calculated based on the motion vector, the feedback control system is formed via the shake correction means. Here, the motion vector detection time delay adds a phase delay element in the feedback control system, causing a deterioration in runout correction performance, and the feedback control system becomes unstable depending on the frequency of the control system, leading to an oscillation state. there is a possibility. Further, in order to reduce the feedback control delay due to the motion vector as much as possible, if the motion vector between two frames calculated from the low resolution image is used for control, the motion vector error is accumulated and the drift detection result has a drift error. Occurs. Patent Document 1 discloses an image pickup device that detects shake applied to an image pickup device using a motion vector and corrects image shake based on the detected runout information. This image pickup apparatus increases the correction gain while ensuring the stability of the feedback control system by attenuating the high frequency component superimposed on the motion vector.

Japanese Unexamined Patent Publication No. 2015-57670

The image pickup apparatus disclosed in Patent Document 1 is based on the premise that the noise component superimposed on the motion vector is a high frequency component having a certain frequency or higher, and depending on how many time counts until the sign inversion of the motion vector is set. Noise reduction performance is determined. Further, when a relatively large noise is always superimposed on the vibration component of the image pickup device detected by the motion vector, it is difficult to strictly separate the noise and the vibration component of the image pickup device. Therefore, the feedback control system becomes unstable due to the influence of high-frequency noise, and a drift error occurs in the runout detection result. An object of the present invention is to provide an image shake correction device capable of performing image shake correction with high accuracy based on a motion vector calculated from a captured image.

The image shake correction device according to the embodiment of the present invention has a first calculation means for calculating a first motion vector in a first cycle from a captured image and a predetermined frame by integrating the first motion vector. An integration means for calculating a predicted value of the movement amount between the images, a second calculation means for calculating a second motion vector in a second cycle longer than the first cycle from the captured image, and a movement amount of the said movement amount. Based on the error calculation means for calculating the error of the predicted value of the movement amount between the predetermined frames based on the predicted value and the second motion vector, and based on the error of the predicted value of the calculated movement amount. The control means for driving the shake correction means used for correcting the image shake that occurs in the captured image is provided.

According to the image shake correction device of the present invention, image shake correction can be performed with high accuracy based on the motion vector calculated from the captured image.

It is a figure which shows the structure of the image pickup apparatus. It is a figure which shows the structure of the image shake correction apparatus. It is a flowchart explaining image shake correction control. This is a flow chart that explains the motion vector calculation process. This is an example of explaining the calculation timing of the motion vector. It is a figure explaining the process of calculating the movement amount error amount. It is a figure explaining the change of the integration time and the discrete frame time.

FIG. 1 is a diagram showing a configuration of an image pickup apparatus having the image shake correction device of the present embodiment.
In the present embodiment, a digital camera will be described as an example as an image pickup device. The image pickup device may have a moving image shooting function. The image pickup apparatus shown in FIG. 1 includes a zoom lens 101 to a control unit 119.

The zoom lens 101 constitutes an imaging optical system. The zoom drive unit 102 drives the zoom lens 101 under the control of the control unit 119, and the zoom lens 101 moves in the optical axis direction of the image pickup optical system to change the focal length. The image shake correction lens 103 is a shake correction means used for correcting image shake that occurs in a captured image. The image shake correction lens 103 is movable in a direction orthogonal to the optical axis of the image pickup optical system, and is driven and controlled by the image shake correction lens driving unit 104.

The aperture / shutter unit 105 is a mechanical shutter having an aperture function. The aperture / shutter drive unit 106 drives the aperture / shutter unit 105 according to the control of the control unit 119. The focus lens 107 constitutes an image pickup optical system and can move forward and backward in the optical axis direction of the image pickup optical system. The focus drive unit 108 drives the focus lens 107 according to the control of the control unit 119. The image pickup unit 109 photoelectrically converts the subject light incident through the image pickup optical system into an electric signal related to the captured image by using an image pickup element such as a CCD image sensor or a CMOS image sensor. CCD is an abbreviation for Charge Coupled Device. CMOS is an abbreviation for Complementary Metal Oxide Semiconducor. The image pickup signal processing unit 110 performs predetermined processing such as A / D conversion, correlation double sampling, gamma correction, white balance correction, and color interpolation processing on the electric signal output from the image pickup unit 109, and performs a video signal. Convert to.

The video signal processing unit 111 performs processing according to the application on the video signal output from the image pickup signal processing unit 110. Specifically, the video signal processing unit 111 generates a video signal for display, encodes for recording, creates a data file, and the like. The display unit 112 displays an image as necessary based on the display video signal output by the video signal processing unit 111. The power supply unit 115 supplies power to the entire image pickup apparatus. The external input / output terminal unit 116 inputs / outputs a communication signal and a video signal to / from an external device. The operation unit 117 has buttons, switches, and the like for giving instructions to the image pickup device by the user, and is configured so that the first switch (SW1) and the second switch (SW2) are turned on in order according to the pressing amount. Includes release button. When the release button is pressed halfway, SW1 is turned on, and when the release button is pressed all the way, SW2 is turned on. The storage unit 118 stores various data including video signals.

The control unit 119 controls the entire image pickup apparatus. The control unit 119 has, for example, a CPU, a ROM, and a RAM, and controls each unit of the image pickup apparatus by expanding and executing a control program stored in the ROM in the RAM. CPU is Central Processing
It is an abbreviation for Unit. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. By controlling the control unit 119, the function of the image shake correction device included in the image pickup device is realized. The configuration of the image shake correction device will be described later with reference to FIG.

Further, the control unit 119 performs AE processing for determining the aperture value and the shutter speed for obtaining an appropriate exposure amount based on the luminance information of the video signal and the predetermined program diagram. When SW1 is turned on, the control unit 119 calculates the AF evaluation value based on the display video signal output by the video signal processing unit 111 to the display unit 112. Then, the control unit 119 controls the focus drive unit 108 based on the AF evaluation value to perform automatic focus adjustment. Further, when the release switch SW2 included in the operation unit 117 is turned on, the control unit 119 takes a picture at a determined aperture and shutter speed, and processes an electric signal read from the image pickup unit 109. Each processing unit is controlled so that the video signal is stored in the storage unit 118.

Further, the operation unit 117 has an image shake correction switch that makes it possible to select on / off of the shake correction mode. When the shake correction mode is turned on by the image shake correction switch, the control unit 119 performs image shake correction control. Specifically, the control unit 119 instructs the image shake correction lens drive unit 104 to perform an image shake correction operation, and the image shake correction lens drive unit 104 receives the instruction until the image shake correction operation is turned off. Performs image shake correction operation. Further, the operation unit 117 has a shooting mode selection switch capable of selecting one of a still image shooting mode and a moving image shooting mode. Through the selection of the shooting mode by operating the shooting mode selection switch, the control unit 119 can change the operating conditions of the image shake correction lens driving unit 104.

Further, the operation unit 117 has a reproduction mode selection switch for selecting a reproduction mode. When the reproduction mode is selected by operating the reproduction mode selection switch, the control unit 119 stops the image shake correction operation. Further, the operation unit 117 includes a magnification change switch for instructing the zoom magnification change. When an instruction to change the zoom magnification is given by operating the magnification change switch, the zoom drive unit 102 that receives the instruction via the control unit 119 drives the zoom lens 101 to the instructed zoom position. To move.

FIG. 2 is a diagram showing a configuration of an image shake correction device of the present embodiment.
The angular velocity sensor 201 detects the runout applied to the image pickup apparatus. The A / D converter 202 converts the output of the angular velocity sensor 201 into a digital signal. The target position calculation unit 203 calculates the target position of the image shake correction lens 103. The position detection sensor 211 detects the position of the image shake correction lens 103. The adder / subtractor 204 subtracts the output of the position detection sensor 211 from the output of the target position calculation unit 203. The control filter 205 calculates a feedback control amount (lens position control amount) that causes the position of the image shake correction lens 103 to follow the target position based on the output of the adder / subtractor 204. The adder / subtractor 206 adds the output of the first feedback gain 207 and the output of the second feedback gain 212 (motion vector control amount) to the output of the control filter 205. As a result, the control amount used for driving the image shake correction lens 103 can be obtained. The adder / subtractor 206 outputs the obtained controlled variable to the image shake correction lens driving unit 104.

The captured image output by the image pickup unit 109 is processed by the image pickup signal processing unit 110 and output to the first motion vector detection unit 210 and the second motion vector detection unit 215. The first motion vector detection unit 210 detects (calculates) the first motion vector from the captured image in the first cycle under the control of the control unit 119. The first motion vector processing unit 209 performs predetermined processing on the output of the first motion vector detection unit 210. The second motion vector detection unit 215 detects (calculates) the second motion vector from the captured image in a second cycle longer than the first cycle under the control of the control unit 119. The second motion vector processing unit 214 performs predetermined processing on the output of the second motion vector detection unit 215. The first feedback gain unit 207 calculates the feedback control amount by multiplying the first motion vector by a predetermined gain. The integrator 208 integrates (integrates) the first motion vector in a predetermined time. As a result, the predicted value of the movement amount in the predetermined time (between the predetermined frames) is calculated. The adder / subtractor 213 functions as an error calculating means, and takes a difference between the output of the second motion vector processing unit 214 and the output of the integrator 208. As a result, the error amount (movement amount error amount) of the predicted value of the movement amount between predetermined frames is calculated. The second feedback gain 212 calculates the feedback control amount by multiplying the calculated movement amount error amount by a predetermined gain. The integration time (integration time) by the integrator 208 and the calculation cycle of the second motion vector by the second motion vector detection unit 215 are changed according to the shooting setting information.

FIG. 3 is a flowchart illustrating image runout correction control in the present embodiment. Further, FIG. 4 is a flow chart for explaining the motion vector calculation process.
The image shake correction control shown in the flowchart of FIG. 3 is repeatedly executed at a predetermined sampling cycle. Further, the calculation process of the motion vector information shown in the flowchart of FIG. 4 is repeatedly executed at the frame rate cycle of the image.

In S101 of FIG. 3, the angular velocity sensor 201 that functions as a shake sensor detects shake applied to the image pickup device due to camera shake or the like. The detected runout information is converted into an electric signal, and the analog signal is converted into a digital signal by the A / D converter 202. Subsequently, in S102, the target position calculation unit 203 calculates the target position of the image shake correction lens 103 for correcting the image shake caused by the shake based on the shake information.

Next, in S103, the position detection sensor 211 detects the position of the image shake correction lens 103. The detected position information is converted from an analog signal to a digital signal by the A / D converter 216. The adder / subtractor 204 subtracts the position information of the image shake correction lens 103 output from the A / D converter 217 from the target position information of the image shake correction lens 103. Then, in S104, the control filter 205 calculates the lens position control amount based on the output of the adder / subtractor 204.

Further, in parallel with the acquisition process of the runout information by the runout sensor, the motion vector information calculation process is executed in S201 to S209 of FIG. Since the motion vector information calculation process is performed based on the difference in pixel positions of a plurality of images by a known method, it is executed depending on the sampling time in which a plurality of images can be acquired. The calculation process of the motion vector information may be executed by sampling different from the acquisition sampling of the runout by the runout sensor.

In S201, the control unit 119 determines whether or not the integrator 208 is instructed to change the integration time of the first motion vector. When the change of the integration time of the first motion vector is instructed, the process proceeds to S202. If the change of the integration time of the first motion vector is not instructed, the process proceeds to S203. In S202, the control unit 119 changes the integration time of the first motion vector, and the process proceeds to S203.

In S203, the control unit 119 determines whether or not the change of the frame interval used for calculating the second motion vector is instructed. When the change of the frame interval used for the calculation of the second motion vector is instructed, the process proceeds to S204. If the change of the frame interval used for calculating the second motion vector is not instructed, the process proceeds to S205.

In S204, the control unit 119 changes the frame interval used for calculating the second motion vector, and the process proceeds to S205. In S205, the control unit 119 determines whether it is time to acquire an image. When it is time to acquire an image, the process proceeds to S206. If it is not the timing to acquire the image, the process ends.

In S206, the image pickup unit 109 converts the subject light into an electric signal according to the instruction of the control unit 119, and acquires the signal related to the captured image. Further, the image pickup signal processing unit 110 converts the signal related to the image pickup image from the analog log signal to the digital signal, and performs predetermined image processing. As a result, an evaluation image for calculating the motion vector is generated.

In S207, the first motion vector detection unit 210 and the first motion vector processing unit 209 compare the image one frame before and the image of the current frame stored in advance, and determine the first motion vector from the deviation of the image. calculate. As a motion vector detection method, a known correlation method, block matching method, or the like is applied. In the present invention, any known method may be used for calculating the motion vector.

Next, in S504, the control unit 119 determines whether or not the calculation timing of the second motion vector has been reached. If it is not the calculation timing of the second motion vector, the process ends. When the calculation timing of the second motion vector is reached, the process proceeds to S209. In S209, the second motion vector detection unit 215 and the second motion vector processing unit 214 calculate the second motion vector.

FIG. 5 is an example for explaining the calculation timing of the first motion vector and the second motion vector.
In FIG. 5, the portion shown by the parallelogram represents the timing of data reading by the image pickup unit 109. Further, in FIG. 5, for reading the frame image shown in Img00 to Img04, the calculation timing of the first motion vector, the calculation timing of the second motion vector, and the image of two frames when calculating the motion vector. Time relationship is shown. The detection of each motion vector is performed by a matching process of two frame images. In the present embodiment, the control unit 119 functions as a first calculation means for calculating the first motion vector in the first cycle from the captured image. Further, the control unit 119 functions as a second calculation means for calculating the second motion vector from the captured image in the second cycle longer than the first cycle. Specifically, the control unit 119 calculates the motion vector from the frame images having different time intervals for the first motion vector and the second motion vector.

The control unit 119 calculates the first motion vector from the time-continuous frame images, for example, Img00 and Img01. On the other hand, the control unit 119 calculates the second motion vector from the frame images separated by one frame in time, such as Img00 and Img02. That is, the control unit 119 calculates the second motion vector at a frame interval longer than the frame interval when calculating the first motion vector. The calculation timing of the motion vector shown in FIG. 5 is an example. The time interval of the frame for calculating the motion vector may be set so that the timing at which the integration (integration) of the first motion vector is completed corresponds to the calculation timing of the second motion vector. The second motion vector may be calculated from a frame image separated by two frames, for example, Img01 and Img04.

Returning to the description of FIG. In S107, the control unit 119 determines whether or not the calculation of the first motion vector is completed. If the calculation of the first motion vector is not completed, the process proceeds to S113, and the previous motion vector control amount is held. When the calculation of the first motion vector is completed, the process proceeds to S108 and S114. That is, the processing of S108 and the processing of S114 are executed in parallel.

In S108, the control unit 119 integrates (integrates) the first motion vector in a predetermined time. As a result, the predicted value of the movement amount between predetermined frames is calculated. In parallel with the processing of S108, in S114, the control unit 119 integrates a predetermined first feedback gain based on the first motion vector. As a result, the feedback control amount based on the first motion vector is calculated. After the processing of S114, the processing proceeds to S112.

In S109, the control unit 119 determines whether the calculation of the second motion vector is completed. If the calculation of the second motion vector is not completed, the process proceeds to S114. When the calculation of the second motion vector is completed, the process proceeds to S110. In S110, the control unit 119 compares the predicted value of the movement amount between predetermined frames calculated in S108 with the movement amount between predetermined frames obtained from the second motion vector, so that the movement amount between predetermined frames is compared. Calculate the error amount (movement amount error amount) of the predicted value of. In S111, the control unit 119 clears 0 the integrated value obtained in S108. Subsequently, in S112, the feedback control amount calculated in S114 and the feedback control amount based on the movement amount error amount calculated in S110 are added to calculate the motion vector control amount. In S105, the control unit 119 adds the motion vector control amount calculated in S112 to the lens position control amount calculated in S104. As a result, the control amount used for driving the image shake correction lens 103 is calculated. Then, the control unit 119 corrects the image shake by driving the image shake correction lens 103 based on the control amount calculated in S112.

FIG. 6 is a diagram illustrating a process of calculating a movement amount error amount.
In the calculation of the feedback control amount by the first feedback gain 207, a time delay of at least one field occurs with respect to the actual runout due to the influence of the image accumulation time and readout time, the matching time between a plurality of frame images, and the like. .. Since the image shake correction device forms a feedback loop via the image shake correction means 103 that optically corrects the image shake, the feedback control system becomes unstable depending on the time delay and the control frequency, resulting in oscillation. There is a possibility that it will be reached.

In order to improve the time delay, it is conceivable to reduce the resolution (image size) of the image used for calculating the first motion vector. However, if the image size is reduced, the resolution of the image information is lowered, so that a detection error occurs in the first motion vector. Therefore, driving the image shake correction lens 103 with the detection information having an error causes a low frequency drift error. In the present embodiment, accurate image shake correction is realized by the method described with reference to FIG.

In FIG. 6, the solid line indicates a first motion vector with an error calculated between each image frame. The amount of movement of the solid lines connected (integrated) over time is the actual amount of movement of the image shake correction lens 103. Therefore, the accumulation of detection errors causes drift errors. For the stability of the feedback control system, it is desired to feed back the motion vector without time delay, but there has been a problem that a drift error occurs in the past. Therefore, in the present embodiment, the control unit 119 calculates the motion vector between the discrete frames as the second motion vector. The discrete frames are not between frames that are continuous in time, but between frames that are separated by the discrete frame time (time for n frames: n is an integer of 1 or more).

The control unit 119 integrates (integrates) the first motion vector with the discrete frame time by the integrator 208. The control unit 119 subtracts the second motion vector from the predicted value of the motion amount between predetermined frames obtained by integrating the first motion vector by the adder / subtractor 213, and the motion amount error amount according to the drift error. Is calculated. Then, the control unit 119 feeds back the calculated movement amount error amount to the lens position control amount. This makes it possible to prevent the image shake correction lens 103 from malfunctioning due to the drift error of the first motion vector.

The amount of movement between frames 0-1 shown by the solid line in FIG. 6 is, for example, the amount of movement corresponding to the motion vector detected between Img00 and Img01 in FIG. The movement amount between frames 1-2 is, for example, the movement amount corresponding to the motion vector detected between Img01 and Img02 in FIG. The movement amount between frames 2-3 is, for example, the movement amount corresponding to the motion vector detected between Img02 and Img03 in FIG. The movement amount between frames 3-4 is, for example, the movement amount corresponding to the motion vector detected between Img03 and Img04 in FIG.

The movement amount shown by the solid line in FIG. 6 is a predicted value of the movement amount between predetermined frames, and includes the original movement amount + the movement amount due to the vector detection error. On the other hand, the second motion vector is directly calculated between frames having the same time interval as the integration time (discrete frame time) of the first motion vector. The movement amount shown by the dotted line in FIG. 6 (movement amount between frames 0-2 and frame 2-4) is a movement amount corresponding to the second motion vector. The amount of movement between frames 0-2 is, for example, the amount of movement between Img00 and Img02 in FIG. The amount of movement between frames 2-4 is, for example, the amount of movement between Img02 and Img04 in FIG.

The control unit 119 subtracts the movement amount between frames 0 and 2 from the predicted value of the movement amount obtained by integrating the movement amount between frames 0-1 and the movement amount between frames 1-2, for example, in FIG. The drift error movement amount shown as the movement amount error between frames 0 and 2 in the middle is calculated. Further, the control unit 119 subtracts the movement amount between frames 2 and 4 from the predicted value of the movement amount obtained by integrating the movement amount between frames 2-3 and the movement amount between frames 3-4, for example. , The movement amount error amount shown as the movement amount error between frames 2-4 in FIG. 6 is calculated. The control unit 119 feeds back the calculated movement amount error amount to the lens position control amount to calculate the control amount used for driving the image shake correction lens 103. As a result, the drift of the image shake correction lens 103 can be reduced and the shake correction performance can be improved.

FIG. 7 is a diagram illustrating changes in the integration time of the first motion vector and the discrete frame time when calculating the second motion vector.
In FIGS. 7A and 7B, the "integration time" indicates the integration time of the first motion vector. "Second motion vector calculation frame interval" indicates the discrete frame time when calculating the second motion vector.

FIG. 7A shows an example of changing the integration time and the discrete frame time according to the frame rate. The control unit 119 changes the integration time of the first motion vector and the discrete frame time used for calculating the second motion vector according to the frame rate of the captured image. Specifically, as shown in FIG. 7A, the control unit 119 increases the integration time and the discrete frame time as the frame rate value increases (the period becomes faster). This is because the evaluation time of the movement amount error amount becomes short unless the integration time is lengthened due to the increase in the frame rate value, and the movement amount error amount cannot be calculated accurately.

FIG. 7B shows an example of changing the integration time and the discrete frame time according to the image resolution. The control unit 119 changes the integration time and the discrete frame time according to the image resolution used for calculating the first motion vector. Specifically, the control unit 119 increases the integration time and the discrete frame time as the image resolution used for calculating the first motion vector becomes higher. As the image resolution becomes higher, the detection error of the first motion vector becomes smaller, and the amount of movement error becomes smaller. Therefore, the control unit 119 increases the integration time and the discrete frame time, reduces the frequency of drift correction, and reduces the processing load for calculating the movement amount error amount. As described above, according to the image shake correction device of this embodiment, the processing load at the time of calculating the movement amount error amount is reduced by changing the integration time and the discrete frame time according to the shooting setting. be able to.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

103 Image runout correction lens 119 Control unit

Claims (10)

The first calculation means for calculating the first motion vector in the first cycle from the captured image, and
An integration means for calculating a predicted value of the amount of movement between predetermined frames by integrating the first motion vector, and an integration means.
A second calculation means for calculating a second motion vector from the captured image in a second cycle longer than the first cycle, and
An error calculation means for calculating an error in the predicted value of the movement amount between the predetermined frames based on the predicted value of the movement amount and the second motion vector.
An image shake correction device comprising: a control means for driving a shake correction means used for correcting an image shake occurring in the captured image based on an error of a predicted value of the calculated movement amount. The control means adds the control amount based on the first motion vector and the control amount based on the error of the predicted value of the movement amount to calculate the control amount used for driving the runout correction means. The image shake correction device according to claim 1. The image shake correction device according to claim 1 or 2, wherein the integration means changes the integration time of the first motion vector according to the frame rate of the captured image. The image shake correction device according to claim 3, wherein the integration means increases the integration time of the first motion vector as the value of the frame rate related to the captured image becomes larger. One of claims 1 to 4, wherein the integration means changes the integration time of the first motion vector according to the resolution of the image used for calculating the first motion vector. Image shake correction device according to. The image shake correction device according to claim 5, wherein the integration means increases the integration time of the first motion vector as the resolution of the image used for calculating the first motion vector becomes higher. .. The second calculation means calculates the second motion vector at a frame interval longer than the frame interval when the first calculation means calculates the first motion vector.
5. The image shake correction device according to 6. The seventh aspect of claim 7, wherein the integration means increases the frame interval when calculating the second motion vector as the resolution of the image used for calculating the first motion vector becomes higher. Image shake correction device. An image pickup apparatus comprising: an image pickup element that photoelectrically converts subject light and outputs a signal related to the captured image, and an image shake correction device according to any one of claims 1 to 8. The process of calculating the first motion vector from the captured image in the first cycle, and
A step of calculating a predicted value of the amount of movement between predetermined frames by integrating the first motion vector, and
A step of calculating a second motion vector from the captured image in a second cycle longer than the first cycle, and
A step of calculating an error in the predicted value of the movement amount between the predetermined frames based on the predicted value of the movement amount and the second motion vector.
Control of the image shake correction device, which comprises a step of driving a shake correction means used for correcting the image shake occurring in the captured image based on the error of the predicted value of the calculated movement amount. Method.

Images