JP3252416B2 - Image stabilization device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image stabilizing apparatus for correcting the image pickup output of a handy type video camera by detecting the amount of movement of an image caused by a so-called camera shake contained in video data.

[0002]

2. Description of the Related Art Generally, in a handy type video camera, camera shake during photographing, that is, camera vibration appears as image vibration. Therefore, as an image stabilization device for correcting an image vibration caused by such a camera shake, for example, as disclosed in Japanese Patent Application Laid-Open No. 63-166370, a motion vector of an image is detected and based on this motion vector. A technique for correcting video data stored in an image memory has been proposed.

For detecting a motion vector of an image, for example, a block matching method is employed. In the detection of a motion vector of an image by the block matching method, a screen is divided into a number of regions (referred to as blocks), and a representative point pixel of a previous field located at the center of each block and a pixel of each pixel in a block of the current field are determined. The absolute value of the field difference from the image data is calculated, the absolute value of the field difference of each block is integrated for each corresponding pixel to obtain a correlation integral value, and the correlation integration having coordinates corresponding to the pixel array of one block Form a value table. Then, the motion vector of the entire screen is determined using the coordinate value of the minimum value of the correlation integral value in the correlation integrated value table as the coordinate value of the motion vector of the image.

[0004] The image blur correction device converts the detected motion vector into a correction signal, and performs a correction for moving the current image using the correction signal. The correction accuracy of such an image stabilization device depends on the detection accuracy of the motion vector of the image.

[0005]

By the way, as described above, the motion vector of the entire screen is determined by using the coordinate value of the minimum value of the correlation integrated value in the correlation integrated value table as the coordinate value of the motion vector of the image. In the detection of a motion vector of an image by the block matching method, a motion vector due to camera shake and a motion vector due to movement of a subject are simultaneously generated. Therefore, the image shake correction apparatus needs to determine the motion vector due to the camera shake and the motion vector due to the movement of the subject, and form a camera shake correction signal using only the motion vector due to the camera shake to perform the camera shake correction. If the motion vector due to the movement of the subject is erroneously determined to be the motion vector due to camera shake and the camera shake correction is performed, the background image that should be stationary moves due to the camera shake correction, which is unnatural. Becomes an image. Conversely, if the motion vector due to the camera shake is erroneously determined to be the motion vector due to the movement of the subject, the camera shake cannot be corrected.

The present invention has been made in view of such circumstances, and has as its object to enable high-performance camera shake correction in a handy-type video camera or the like.
It is an object of the present invention to provide an image stabilization apparatus for an image, in which it is reliably determined whether a detected motion vector of an image is a motion vector of an image due to camera shake or a motion vector of a movement of a subject, thereby performing camera shake correction. .

[0007]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, an image stabilizing apparatus for an image according to the present invention is a representative image for each block obtained by dividing a fixed unit image formed by an input video signal into a plurality of blocks. The image data I _k (0,0) of the point pixel is stored in the memory for a fixed unit period, and the image data I _k (x, y) of each pixel of the block of the current field and the representative of the block of the previous field read from the memory are stored. The absolute value | I _k-1 of the difference from the image data I _k-1 (0,0) of the point pixel
A motion vector detecting unit that detects a motion vector of an image based on (0,0) −I _k (x, y) |, and for each block obtained by dividing a fixed unit image composed of the input video signal into a plurality of blocks , The image data I _k (x
₀ , y ₀ ) is stored in the memory for a fixed unit period, and the image data I _k (x, y) of each pixel of the current field block and the image data I _k of each representative point pixel of the previous field block read from the memory are read out. _a camera shake determination unit that determines whether or not the motion vector detected by the motion vector detection unit is due to a camera shake based on an inter-field correlation with _k−1 (x ₀ , y ₀ ); A correction amount generation unit that forms a camera shake correction signal of a correction amount corresponding to the motion vector determined by the camera shake determination unit to be caused by camera shake, and a camera shake correction supplied from the correction amount generation unit A correction unit for performing a camera shake correction process on an input video signal by a signal.

[0008]

In the image stabilization apparatus for an image according to the present invention, a motion vector of an image is obtained by block matching using image data of a representative point pixel for each block obtained by dividing a fixed unit image formed of an input video signal into a plurality of blocks. Is detected by a motion vector detection unit, and the motion vector is determined based on the inter-field correlation of each image data of a plurality of representative point pixels for each block obtained by dividing a fixed unit image formed of the input video signal into a plurality of blocks. Is determined by the camera shake determiner whether or not is caused by camera shake, a camera shake correction signal of a correction amount corresponding to a motion vector caused by camera shake is formed by the correction amount generator, and the input video signal is Is subjected to camera shake correction processing.

[0009]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram of an image stabilizing apparatus according to an embodiment of the present invention; An image stabilization apparatus for an image according to the present invention is configured, for example, as shown in FIG.

The image blur correction device shown in FIG. 1 is a device in which the present invention is applied to a camera shake correction device for correcting the motion of an image due to a camera shake in a handy type video camera. The camera includes a camera shake determination unit 20, a correction amount generation unit 30, and a correction unit 40. In FIG. 1, input video data obtained by digitizing a video signal obtained as an imaging output by an imaging unit (not shown) of the video camera is supplied to a signal input terminal 1.

In this camera shake correction apparatus, the motion vector detecting section 10 includes a representative point memory 11 and a subtraction circuit 12 to which the input video data is supplied via the signal input terminal 1, and a subtraction output by the subtraction circuit 12. The circuit comprises a correlation integrated value table forming circuit 13 to which data is supplied, and a motion vector detecting circuit 14 to which a correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming circuit 13 is supplied.

The representative point memory 11 in the motion vector detecting unit 10 is composed of input video data.
The image data I _k (0,0) of the representative point pixel for each block obtained by dividing the image of the field into a plurality is stored. Specifically, for example, as shown in FIG. 2, a screen of one field is divided into blocks of m pixels × n lines, and as shown in FIG. 3, a pixel S (0,0) at the center of each block is set as a representative point. age,
The image data I _k (0,0) of each representative point pixel is stored in the representative point memory 11 for one field period. Note that the representative points are uniformly distributed on the screen. Then, the image data I _k−1 (0,0) of each representative point pixel one field before read from the representative point memory 11 is supplied to the subtraction circuit 12.

The subtraction circuit 12 is connected to the signal input terminal 1
, The image data of the current field, i.e., the image data I _k (x, y) of each pixel of m × n and the representative point memory 1
1 and the difference from the image data I _k-1 (0,0) of the representative point pixel of the corresponding block of the previous field, that is, the absolute value of the difference between the fields | I _k-1 (0,0) -I _k
(X, y) |. Then, the field difference absolute value obtained as the subtraction output data by the subtraction circuit 12 is supplied to the correlation integrated value table forming circuit 13.

The correlation integrated value table forming circuit 13 corresponds to the field difference absolute value | I _k-1 (0,0) -I _k (x, y) | of each block obtained by the subtraction circuit 12. The pixels are integrated over one field period to form a correlation integrated value table having m × n integer coordinates corresponding to the pixel arrangement of one block. This correlation integrated value table forming circuit 13
The correlation integrated value table formed by the following formula indicates the integrated value of m × n field difference absolute values | I _k-1 (0,0) −I _k (x, y) | The correlation integrated value of the coordinates having the strongest field correlation becomes the minimum value. Then, m × n correlation integrated values in the correlation integrated value table formed by the correlation integrated value table forming circuit 13 are supplied to the motion vector detecting circuit 14.

The motion vector detection circuit 14 detects the coordinates of the minimum value of the correlation integrated value in the correlation integrated value table formed by the correlation integrated value table forming circuit 13. The correlation integrated value of the correlation integrated value table formed by the correlation integrated value table forming circuit 13 is the image data I _k−1 of the representative point pixel of each block.
This indicates the inter-field correlation between (0, 0) and the image data I _k (x, y) of another pixel. The coordinates corresponding to a pixel having a higher correlation have a smaller value, and the correlation of the coordinates corresponding to a motion vector is smaller. The integrated value becomes the minimum value. That is, the motion vector detection circuit 14 detects the coordinate of the minimum value of the correlation integrated value in the correlation integrated value table as the coordinate indicating the motion vector.

The coordinates of the minimum value of the correlation integrated value, that is, the motion vector detected by the motion vector detection circuit 14 are supplied to the camera shake determination section 20 and the correction amount generation section 30.

In this camera shake correction apparatus, the camera shake determination section 20 includes a representative point memory 21 to which the input video data is supplied via the signal input terminal 1, a subtraction circuit 22, and a condition determination circuit 23. Condition judgment circuit 2
3, a frequency distribution table forming circuit 24 that forms a frequency distribution table based on the determination output, and a process of determining whether the motion vector of the image detected by the motion vector detection unit 10 is a motion vector of an image due to camera shake. Is performed on the basis of the frequency distribution table formed by the frequency distribution table forming unit 24.

In the camera shake judging section 20, the representative point memory 21 stores a plurality of representative point pixels for each block obtained by dividing the image of one field constituted by the input video signal into a plurality of blocks [for example, as shown in FIG. , Nine representative point pixels S (0, 0,
_{0), S (+ Δ x} , 0), S (+ Δ x, -Δ y), S
(0, −Δ _y ) _, S (−Δ _x , −Δ _y ) _, S (−Δ _x ,
_{0), S (-Δ x,} + Δ y), S (0, + Δ y), S
(+ Δ _x , + Δ _y )]] of each image data I _k (x ₀ ,
y ₀ ) is stored for one field period. Each of the blocks corresponds to each of the blocks obtained by dividing the image of one field into a plurality in the camera shake determination unit 20. Then, the image data I _k-1 (x ₀ , y ₀ ) of each representative point pixel one field before is read from the representative point memory 21 and supplied to the subtraction circuit 22.

The subtraction circuit 22 is connected to the signal input terminal 1
, The image data of the current field, the image data I _k (x, y) of each of the m × n pixels for each block and the representatives of the previous field read from the representative point memory 21. Absolute value | I of difference from image data I _k-1 (x ₀ , y ₀ ) of point pixel
_{k-1 (x 0, y} 0) -I k (x, y) | that is, detecting the absolute value of the inter-field difference. And this subtraction circuit 2
The field difference absolute value obtained as the subtraction output data by 2 is supplied to the condition determination circuit 23.

The condition judging circuit 23 determines the representative data located at the center of each block for the image data I _k-1 (x ₀ , y ₀ ) of each representative point pixel one field before, which is read from the representative point memory 21. The image data I _k-1 (0,0) of the point pixel S (0,0) and the image data I _k of another representative point pixel
_k-1 (x ₀ , y ₀ ) Difference absolute value | I _k-1 (0,0)
-I _k-1 (x ₀ , y ₀ ) | is the number of representative point pixels satisfying the determination condition of | I _k-1 (0,0) -I _k-1 (x ₀ , y ₀ ) | ≧ Th ₁ by detecting the N ₁ for each block, the number N ₁ detects a block corresponding to the lower flat image first determination condition is satisfied activities that N ₁ ≧ NTH _1.

Then, for a block corresponding to a flat image having low activity satisfying the first determination condition, the field difference absolute value | I _k-1 (x ₀ , y) of each block detected by the subtraction circuit 22 ₀ ) -I
_k (x, y) | _{is, | I k-1 (x} 0, y 0) -I k (x 0, y 0) | a ≧ Th ₂ comprising determination satisfies the representative point pixel number N ₂ for each block A block having a strong inter-field correlation whose number N ₂ satisfies the second determination condition of N ₂ ≧ NTh ₂ is detected.

Further, the image data I _k (x, x) of the current field of the block satisfying the first and second determination conditions
y), the image data I _k (0,0) of the representative point pixel S (0,0) located at the center of each block and the image data I _k (x ₀ , y ₀ ) of the other representative point pixels Difference absolute value ｜
_{I k (0,0) -I k (} x 0, y 0) | _{is, | I k (0,0) -I} k (x 0, y 0) | ≧ Th 2 comprising determination satisfies representative point pixels The number N ₃ is detected for each block, and a block corresponding to a flat image with low activity that satisfies the third determination condition where the number N ₃ satisfies N ₃ ≧ NTh ₃ is detected as a still block.

For the still blocks satisfying all the first to third determination conditions, the field difference absolute value | I of each block detected by the subtraction circuit 22 is used.
_k−1 (x ₀ , y ₀ ) −I _k (x, y) | is a fourth determination condition of | I _k−1 (0, 0) −I _k (x, y) | ≧ Th ₄ It is determined whether or not the condition is satisfied. Here, the Th ₄ is a threshold for determining the presence or absence of level difference in space.

That is, the condition determination circuit 23 detects a still block candidate having a representative point pixel having a strong field correlation according to the first to third determination conditions, and determines the still block candidate according to the fourth determination condition. I _k of each pixel with respect to the image data I _k of representative point pixels (0,0)
Stillness determination based on the correlation of (x, y) is performed.

The output of the judgment by the condition judgment circuit 23 is supplied to the frequency distribution table forming circuit 24.

The frequency distribution table forming circuit 24 is based on the memory table 24A having coordinates corresponding to the pixel arrangement of one block and the output of the first and second determination conditions based on the determination output by the condition determination circuit 23. And an adder 24B that increments the frequency f (x, y) of the corresponding coordinate every time a pixel satisfying the following condition is detected. In the frequency distribution table forming circuit 24, the image data I
_k (x, y) and the absolute value | I _k-1 (0,0) of the difference between the image data I _k-1 (0,0) of each representative point pixel one field before.
0) -I _k (x, y) |, a conditional frequency distribution table showing the correlation between the one field of the representative point pixels is formed in the memory table 24A. Then, the frequency value of the frequency distribution table indicating the correlation between one field formed in the memory table 24A by the frequency distribution table forming circuit 24 is supplied to the camera shake determination circuit 25.

The camera shake determination circuit 25 calculates the frequency distribution table formed by the frequency distribution table forming circuit 24,
If the frequency value of the coordinates corresponding to the motion vector of the image detected by the motion vector detection unit 10 is smaller than a predetermined value, the motion vector of the image detected by the motion vector detection unit 10 is used as the image shake. If the frequency value is larger than a predetermined value, it is determined that the motion vector of the image detected by the motion vector detection unit 10 is not due to camera shake of the image.

Here, for example, as shown in FIG.
When the flat subject OB independently moves by v ₁ in the image with the motion vector V _k due to camera shake between the field and the k field, the image data I _k−1 (0,0) of the central representative point pixel of the block A ), I _k (0,0) have a high inter-field correlation, and the image data I _k (x, y) at the coordinates of the motion vector V _k and the image data I _k−1 (0,0) of the representative point pixel in the previous field ) Is large,
Only by determining the inter-field correlation of the image data of the central representative point pixel, the condition determination circuit 23 determines that the pixel is a significant pixel satisfying the fourth determination condition, and in the frequency distribution table forming circuit 24, the memory table 24A
The frequency f (x, y) of the coordinates of (1) is incremented, and the camera shake determination circuit 25 may make an erroneous determination. On the other hand, in the image stabilizing apparatus of this embodiment,
Since the inter-field correlation of each image data of a plurality of representative point pixels is determined for each block, the camera shake determination unit 20 reliably determines whether the motion vector is caused by camera shake. be able to.

Then, the judgment output of the camera shake judging section 20 is supplied to the correction amount generating section 30.

When the determination output indicating that the motion vector detected by the motion vector detection unit 10 is caused by the camera shake of the image is supplied from the camera shake determination unit 30, the correction amount generation unit 30 Motion vector detector 1
The motion vector detected by the motion vector 0 is represented by a shake vector V _t '.
As to form the image stabilization signal _{X t = X t-1 -V} t ' becomes the correction amount X _t, and supplies the image stabilization signal to the correcting unit 30. When the determination output indicating that the motion vector detected by the motion vector detection unit 10 is not caused by the camera shake of the image is supplied from the camera shake determination unit 30, the correction amount generation unit 30 _t 'is replaced by the zero vector V [0,0]
Is formed, and the camera shake correction signal is supplied to the correction unit 40.

As shown in FIG. 6, for example, the correction section 40 includes an address control circuit 41 and a select signal generation circuit 42 to which a camera shake correction signal is supplied from the correction amount generation section 30, and an address control circuit 41. Memory 43 and peripheral memory 44 in which video data is written / read in accordance with an address signal supplied from
And a selector 45 for selectively outputting video data read from the field memory 43 and the peripheral memory 44 in accordance with a select signal supplied from the select signal generating circuit 42.

Input video data supplied via the signal input terminal 1 is sequentially written into the field memory 43. The read address of the field memory 43 is controlled by the camera shake correction signal in accordance with the camera shake vector. As a result, video data obtained by moving one field of input video data in accordance with the camera shake vector is obtained from the field memory 43. Then, the video data read from the field memory 43 and the peripheral video data read from the peripheral memory 44 are combined by selection by the selector 45, and output from the signal output terminal 2 as video data subjected to camera shake correction processing.

In the peripheral memory 44, video data of a peripheral portion corresponding to a correction range of an image based on video data subjected to camera shake correction output via the selector 45 is sequentially written as peripheral video data.

In this camera shake correction apparatus, the camera shake determination section 20 reliably determines whether or not the motion vector of the image detected by the motion vector detection section 10 is attributable to camera shake as described above. Since the correction can be performed, camera shake correction can be reliably performed, and a natural image output can be obtained.

[0035]

As is apparent from the above description, in the image stabilization apparatus for an image according to the present invention, a representative point pixel of each block obtained by dividing a fixed unit image formed by an input video signal into a plurality of blocks is divided. A motion vector detection unit detects a motion vector of an image by block matching using image data, and a plurality of representative point pixels for each block obtained by dividing a fixed unit image formed by the input video signal into a plurality of blocks. Whether or not the motion vector is due to camera shake can be reliably determined by the camera shake determination unit based on the inter-field correlation of the data. Is formed by the correction amount generation unit, and the correction unit can reliably perform the camera shake correction processing on the input video signal.

As described above, in the image stabilizing apparatus for an image according to the present invention, it is possible to reliably determine whether or not a motion vector of an image is caused by a camera shake and to perform a camera shake correction. High-performance camera shake correction in a video camera or the like can be performed.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a camera shake correction apparatus according to the present invention.

FIG. 2 is a diagram illustrating a state in which a screen is divided into blocks in a motion vector detection unit of the camera shake correction apparatus.

FIG. 3 is a diagram showing a structure of one block of the screen divided into blocks.

FIG. 4 is a diagram schematically illustrating a setting state of a plurality of representative point pixels set for each block.

FIG. 5 is a diagram schematically showing a state of generation of a motion vector due to a motion of a subject.

FIG. 6 is a block diagram illustrating a configuration of a correction unit of the camera shake correction device.

[Explanation of symbols]

 Reference numeral 10: Motion vector detecting section 11: Representative point memory 12: Subtraction circuit 13: Correlation Integrated value table forming circuit 14 Motion vector detecting circuit 20 Camera shake determination unit 21 Representative point memory 22 ················································································ ········· Correction amount generation unit 40 ········ Correction unit

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Ken Satoshi 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Tsukasa Hashino 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo (56) References JP-A-4-255178 (JP, A) JP-A-4-302590 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) H04N 5/232 G03B 5/00 G06T 7/20 100

Claims (1)

(57) [Claims] An image data I _k (0,0) of a representative point pixel for each block obtained by dividing a fixed unit image composed of an input video signal into a plurality of blocks is stored in a memory for a fixed unit period,
Image data I _k of each pixel of the block of the current field
(X, y) and image data I _k-1 (0, 0) of the representative point pixel of the block of the previous field read from the memory
| I _k-1 (0,0) -I _k (x, y)
And a motion vector detecting unit that detects a motion vector of an image based on |, a plurality of representative point pixel image data I _k (x ₀ , y ₀ ) are stored in the memory for a certain unit period,
Image data I _k of each pixel of the block of the current field
(X, y) and the image data I _k-1 (x ₀ ,
y ₀ ), a camera shake determination unit that determines whether the motion vector detected by the motion vector detection unit is caused by camera shake, A correction amount generator that forms a camera shake correction signal of a correction amount corresponding to the motion vector determined to be caused by camera shake; and a camera shake to an input video signal by a camera shake correction signal supplied from the correction amount generator. An image stabilization device for an image, comprising: a correction unit for performing a correction process.