JP6548409B2 - Image processing apparatus, control method therefor, control program, and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus, a control method thereof, a control program, and an imaging apparatus, and more particularly, to an anti-vibration processing technique for reducing blurring of an image caused by movement (shake) of the imaging apparatus.

  Generally, electronic image stabilization processing is performed to reduce blurring of an image caused by blurring of an imaging device such as a digital still camera or digital video camera. Then, in order to perform the image stabilization processing, it is necessary to detect the amount of movement between frames and to perform the alignment processing of a plurality of images.

  In the method of detecting the amount of movement between frames, for example, shake information obtained by an external device such as a gyro sensor is used. Furthermore, it is also performed to estimate the amount of movement from an image obtained as a result of shooting. For example, as a method of estimating the amount of motion from an image, there is a method of detecting a motion vector using template matching.

  In template matching, for example, one of two images is an original image, and the other is a reference image. A rectangular area of a predetermined size arranged in the original image is used as a template block, and the correlation with the luminance value distribution of the template block is determined in the reference image. Then, a region where the correlation value is the highest in the reference image is set as the movement destination of the template block, and the direction and movement amount with respect to the movement destination when the position of the template block in the original image is based

  Next, statistical processing or the like is performed using a plurality of motion vectors obtained as described above, and motion between images is obtained as a geometric deformation amount. At this time, if a large number of motion vectors can be accurately obtained from the entire image, the geometric deformation amount between the images can be accurately obtained.

  However, it is difficult to obtain a large number of motion vectors with high accuracy in order to reduce processing time and resources.

  On the other hand, if the images are unclear (that is, unclear), it is difficult to obtain the correlation between the images, and the motion vector accuracy is degraded. Here, the unclear image refers to an image in which high frequency components such as edges are attenuated.

  For example, in the case of shooting in a dark place such as at night, in order to increase the amount of light incident on the imaging sensor, shooting may be performed with a slow shutter speed. In such a case, motion blur occurs in the image obtained as a result of shooting, resulting in an unclear image. This reduces the accuracy of the motion vector. Therefore, it is necessary to control photographing so that the accuracy of the motion vector can be guaranteed even if the shutter speed is reduced.

  In order to guarantee the accuracy of the motion vector, for example, there is a method of changing parameters used when detecting the motion vector according to the accuracy of the motion vector. For example, some influence parameters are generated to indicate what kind of influence the motion vector misdetection has on the image, and the reduction ratios of the original image and the reference image are controlled according to the influence parameters. Patent Document 1).

JP, 2010-252259, A

  However, in the method described in Patent Document 1 described above, the reduction ratio of the image is changed based on the dispersion value and the average value of the luminance values in the template block, and the reduction ratio of the image is adjusted according to the shutter speed. It has not been done. Therefore, in the method described in Patent Document 1, depending on the shutter speed, it may not be possible to detect a motion vector with high accuracy.

  Therefore, an object of the present invention is to provide an image processing apparatus capable of accurately detecting a motion vector regardless of the shutter speed, a control method thereof, a control program, and an imaging apparatus.

In order to achieve the above object, the image processing apparatus according to the present invention uses one image as an original image and an image obtained as a result of photographing after the original image as a reference image, the original image and the reference image The reduction means for reduction processing, the determination means for determining the reduction ratio at the time of reduction processing by the reduction means according to the shutter speed at the time of photographing the original image and the reference image, The original for which the reduction processing was performed possess a detecting means for detecting a motion vector between said reference image and said original image based on the image and the reference image, wherein the determination means includes, as the image size increases as the shutter speed becomes slower Determining the reduction ratio .

The control method of the image processing apparatus according to the present invention is a reduction step of processing one original image and one original image and the image obtained as a result of photographing after the original image as a reference image and reducing the original image and the reference image. And a determination step of determining a reduction ratio at the time of performing the reduction process in the reduction step according to a shutter speed at the time of photographing the original image and the reference image, and the original image and the reference image subjected to the reduction process. have a, a detection step of detecting a motion vector between said reference image and the original image on the basis of determination in the determination step, the reduction ratio so that the image size increases as the shutter speed becomes slower It is characterized by

The control program according to the present invention is a control program used in an image processing apparatus, and a computer provided in the image processing apparatus uses one image as an original image and is obtained as a result of shooting after the original image. Using an image as a reference image, a reduction step for reducing the original image and the reference image, and a reduction ratio at the time of performing the reduction processing in the reduction step according to the shutter speed at the time of shooting the original image and the reference image Performing a determining step of determining, and a detecting step of detecting a motion vector between the original image and the reference image based on the original image and the reference image on which the reduction processing has been performed , in decision step, to said determining said reduction ratio so that the image size increases as the shutter speed becomes slower .

  According to the present invention, since the reduction ratio of the image used for detecting the motion vector is determined according to the shutter speed, the motion vector is accurately detected regardless of the shutter speed, and the blurring in the image is favorably made. It can be corrected.

It is a block diagram showing the composition about an example of an imaging device provided with an image processing device by a 1st embodiment of the present invention. It is a flowchart for demonstrating an example of the imaging | photography process performed with the camera shown in FIG. It is a figure which shows the example about the relationship between the reduction ratio of an image, and the detection number of motion vectors. It is a figure which shows the example about the relationship between the reduction ratio of an image, and the precision of a motion vector. It is a figure which shows the relationship between the reduction ratio based on FIG. 3 and FIG. 4, a residual motion vector number, and vector precision. It is a figure for demonstrating an example of the relationship between shutter speed and a reduction ratio, (a) is a figure which shows a 1st example, (b) is a figure which shows a 2nd example. It is a figure for demonstrating an example of a photography scene, (a) is a figure which shows the image in case shutter speed is quick, (b) is a figure which shows the image in case shutter speed is slow. It is a figure for demonstrating the other example of the relationship between shutter speed and a reduction ratio. It is a figure for demonstrating the template matching performed by the motion vector detection circuit shown in FIG. 1, (a) is a figure which shows an example of an original image, (b) is a figure which shows an example of a reference image. It is a block diagram showing the composition about an example of the camera by a 2nd embodiment of the present invention. It is a flowchart for demonstrating an example of the imaging | photography process performed with the camera shown in FIG.

  Hereinafter, an example of an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

First Embodiment
FIG. 1 is a block diagram showing a configuration of an example of an imaging apparatus provided with an image processing apparatus according to a first embodiment of the present invention.

  The illustrated imaging apparatus is, for example, a digital camera (hereinafter simply referred to as a camera), and includes an imaging optical system (hereinafter simply referred to as an optical system). A subject image (optical image) is formed on the image sensor 102 via the optical system 101. The imaging device 102 is a CCD sensor or a CMOS sensor, and outputs an electrical signal (analog signal) according to an optical image by photoelectric conversion.

  The camera signal processing circuit 103 generates an image signal (also referred to as a camera signal) according to an analog signal which is an output of the imaging element 102. The camera signal processing circuit 103 includes, for example, an A / D conversion circuit, an automatic gain control circuit (AGC), and an automatic white balance circuit (AWB), and performs predetermined image processing on an analog signal to perform digital processing. It outputs a camera signal which is a signal.

  The camera signal which is the output of the camera signal processing circuit 103 is temporarily recorded in the memory 104. The memory 104 is a memory for recording one frame or a plurality of frames of the camera signal. In the following description, an image of one frame is referred to as a frame image. Further, in the illustrated example, the imaging device 102 and the camera signal processing circuit 103 constitute an imaging system for acquiring an image.

  The reduction ratio determination circuit 106 determines the reduction ratio when reducing a frame image according to the shutter speed at the time of shooting under the control of the microcomputer 112. Each of the image reduction circuits 107 and 108 performs reduction processing on the frame image recorded in the memory 104 based on the reduction ratio determined by the reduction ratio determination circuit 106.

  The motion vector detection circuit 109 detects a motion vector between two frame images after reduction, which is the output of the image reduction circuits 107 and 108. The geometric deformation parameter estimation circuit 110 uses the motion vector obtained by the motion vector detection circuit 109 to output a correction amount of blurring between frame images as a geometric deformation parameter. The geometric deformation circuit 111 performs geometric deformation processing for correcting blurring of a frame image based on geometric deformation parameters. The microcomputer 112 controls the entire camera.

  FIG. 2 is a flowchart for explaining an example of the photographing process performed by the camera shown in FIG. The processing according to the flowchart in the drawing is performed under the control of the microcomputer 112.

  When shooting is started, under control of the microcomputer 112, the camera signal processing circuit 103 performs image processing of processing on an analog signal that is an output of the imaging element 102 to generate a camera signal (image input: step) S201). Here, the camera signal processing circuit 103 outputs, for example, a 12-bit digital signal as a camera signal. At this time, the camera signal processing circuit 103 performs signal level correction and white level correction by AGC and AWB, and records a camera signal (that is, a frame image) in the memory 104.

  In the illustrated camera, frame images are sequentially generated at a predetermined frame rate, and the frame images are recorded in the memory 104. Then, the same size frame image recorded in the memory 104 is input to the image reduction circuits 107 and 108. The frame images recorded in the memory 104 are sequentially updated.

  Subsequently, under the control of the microcomputer 112, the reduction ratio determination circuit 106 determines the reduction ratio of the frame image as described later, and sets the reduction ratio in the image reduction circuits 107 and 108 (step S202). .

  FIG. 3 is a diagram showing an example of the relationship between the reduction ratio of an image and the number of detected motion vectors.

  In FIG. 3, the horizontal axis indicates the reduction ratio. The larger the reduction ratio, the larger the image size after reduction, and the smaller the reduction ratio, the smaller the image size after reduction. Then, at the reduction ratio = 1, the image is not reduced. The vertical axis indicates the number of motion vectors (remaining motion vector numbers) determined to have high reliability in the detected motion vectors.

  Here, the reliability of the motion vector is determined, for example, according to what texture the motion vector is detected from the region. That is, since the low contrast area and the area including the repetitive pattern are areas where it is difficult to detect a motion vector by template matching, there is a low possibility that a motion vector is accurately detected in the area.

  In order to make a texture determination, the average value and the dispersion value of the luminance values of the pixels in the template are determined, and it is determined whether the area is a low contrast area or a repetitive pattern area according to the average value and the dispersion value. Do. When it is determined that the region is a low contrast region or a repetitive pattern region, motion vectors detected in that region are excluded as having low reliability. Such determination is performed for all motion vectors, and the number of motion vectors with high reliability finally remaining is made the number of remaining motion vectors.

As shown in FIG. 3 , as the reduction ratio is closer to 1, that is, as the size of the image used for detecting the motion vector is larger, there is a tendency that a plurality of locations having high correlation with the template will be generated. It will be detected more often. Then, the motion vector that is erroneously detected can not be used because it causes an error.

  Furthermore, as the reduction ratio is closer to 1, the texture in the template becomes poor, and as a result of being judged as a low contrast region, the number of motion vectors to be excluded increases. Therefore, as the reduction ratio is closer to 1, the number of residual motion vectors decreases.

  On the other hand, as the reduction ratio is closer to 0, that is, as the size of the image used to detect a motion vector is smaller, textures in a wide area of the image will be included in the template. This reduces the low contrast and the motion vectors excluded by the repeated determination, and increases the number of residual motion vectors.

  FIG. 4 is a diagram showing an example of the relationship between the reduction ratio of the image and the accuracy of the motion vector.

  In FIG. 4, the horizontal axis indicates the reduction rate, and the vertical axis indicates the motion vector detection accuracy. As illustrated, as the reduction ratio is closer to 1, that is, as the image size used to detect a motion vector is larger, the accuracy of the motion vector is improved because the image is clear. On the other hand, as the reduction ratio is closer to 0, the image is blurred due to the low pass filter effect of the reduction processing, and the detection accuracy of the motion vector is lowered.

  Therefore, when reducing the image, there is a limit to making the reduction ratio close to 0, and even if the detection accuracy of the motion vector is reduced, the reduction ratio restriction 501 can be maintained so that the accuracy beyond the detection limit can be maintained. Is set. The detection limit is the limit of accuracy that can be regarded as the same as an accurate motion vector. For example, the detection limit is a detection accuracy that does not allow visual recognition of the difference between an image that has been geometrically deformed using an accurate motion vector and an image that has been geometrically deformed using a certain motion vector. Say

  In addition, when obtaining geometric deformation parameters, it is necessary to estimate not only the movement of the translational component but also the movement of the rotational component and the tilting component. For this purpose, it is necessary to detect many motion vectors from the entire image. Therefore, here, the state in which the reduction ratio is close to 0 is taken as the standard state.

  FIG. 5 is a diagram showing the relationship between the reduction ratio, the number of remaining motion vectors, and the vector accuracy based on FIGS. 3 and 4.

  As shown in FIG. 5, when it is desired to detect a large number of motion vectors with standard accuracy (that is, when the number of residual motion vectors is increased), the reduction ratio approaches zero. On the contrary, if it is desired to detect even a small number of motion vectors with high accuracy (that is, if the number of remaining motion vectors is at least good), the reduction ratio will be close to 1. That is, the image size is changed according to the detection accuracy of the motion vector.

  FIG. 6 is a diagram for explaining an example of the relationship between the shutter speed and the reduction ratio. 6 (a) is a diagram showing a first example, and FIG. 6 (b) is a diagram showing a second example.

  In FIG. 6, the horizontal axis indicates the shutter speed, and when the shutter speed is slowed, the camera shake (i.e., shake) increases. The vertical axis indicates the reduction ratio, and the image size increases as the reduction ratio increases.

  In the example shown in FIG. 6A, the relationship between the shutter speed and the reduction ratio changes linearly (linearly), and the reduction ratio approaches 1 as the shutter speed decreases. That is, the image size is increased as the shutter speed decreases. In the example shown in FIG. 6B, the relationship between the shutter speed and the reduction ratio changes exponentially, and here too, the reduction ratio approaches 1 as the shutter speed decreases.

  When the relationship shown in FIG. 6 is used, when the reduction ratio is close to 1, that is, the size of the image used to detect the motion vector is large, the motion vector is detected with a clear image. For this reason, while the accuracy of the detected motion vector is improved, the tendency that a plurality of portions having high correlation with the template are generated is increased, and erroneous detection of the motion vector is likely to occur. And an incorrect motion vector can not be used because it causes an error.

  Furthermore, when motion vectors are detected in a clear image, the texture in the template becomes scarce, and motion vectors that are determined to be low contrast regions and excluded are increased. For this reason, the number of residual motion vectors decreases.

  On the other hand, when the reduction ratio is close to 0, that is, the size of the image used for detection of motion vectors is small, vector detection is performed on an unclear image. Therefore, while the accuracy of the detected motion vector is lowered, the texture in a wide area of the image is included in the template, so that it is difficult to be excluded by low contrast and repetitive pattern determination. As a result, the number of residual motion vectors increases.

  FIG. 7 is a diagram for explaining an example of a shooting scene. 7A shows an image when the shutter speed is fast, and FIG. 7B shows an image when the shutter speed is slow.

  Here, regardless of the shutter speed, the movement (shake) of the object to be photographed and the camera is assumed to be the same. In the example shown in FIG. 7B, since the shutter speed is slower compared to the example shown in FIG. 7A, blurring is likely to occur in the image even if the movement of the camera is similar.

  Since the shutter speed at the time of obtaining the image shown in FIG. 7A is fast, the image is clear. Therefore, there is a strong tendency to generate a plurality of places where the correlation of the template is high. Therefore, when detecting a motion vector in the image shown in FIG. 7A, if the reduction ratio is made close to 0, that is, if the image size used for detecting the motion vector is reduced, the number of remaining motion vectors is improved. The vibration isolation processing can be performed well (standard state).

On the other hand, since the shutter speed at the time of obtaining the image shown in FIG. 7B is slow, the image becomes unclear due to the movement (shake) of the camera. Therefore, if the image is reduced at a reduction rate similar to that in the standard state, the image becomes more unclear and the accuracy of the motion vector becomes less than the accuracy of the motion vector obtained in the standard state. Therefore, for the image shown in FIG. 7B, the motion vector is reduced with the reduction ratio close to 1, that is, the image size used for detecting the motion vector is increased to suppress the image from becoming unclear. To detect Thus, to allow the detection of the motion vector in the standard state and same accuracy.

  However, as described with reference to FIG. 3, if the image size is increased, the number of remaining motion vectors is decreased, and as described above, it is necessary to change the reduction ratio in accordance with the shutter speed.

  If the motion vector obtained as described above is used, it becomes possible to estimate the geometric deformation parameter that accurately indicates the blurring that occurs in the image obtained as a result of the photographing, and the image stabilization processing for the image with high accuracy. Can be applied.

  The shutter speed is different for each mode such as the program AE mode, the aperture priority AE mode, the shutter priority AE mode, and the manual mode. For example, in the case of the program AE mode and the aperture priority AE mode, an appropriate shutter speed is set by the camera. In the program AE mode, the camera automatically sets the shutter speed together with the aperture according to the brightness of the subject. Further, in the case of the aperture priority AE mode, after the aperture is determined, the shutter speed is automatically set so as to obtain a proper exposure.

  On the other hand, in the case of the manual mode or the shutter priority AE mode, the shutter speed is arbitrarily set by the photographer. In the manual mode, the photographer determines both the aperture and the shutter speed. However, in the shutter priority AE mode, after the photographer determines the shutter speed, the camera sets the aperture so as to obtain a proper exposure.

The relationship between the shutter speed and the reduction ratio is not limited to the example shown in FIG. For example, when the shutter speed is set to be slower than "1 / focal length" seconds, the image is likely to be blurred.

  FIG. 8 is a diagram for explaining another example of the relationship between the shutter speed and the reduction ratio.

  In FIG. 8, when the shutter speed is “1 / focal length” or less, blurring due to the movement of the camera hardly occurs in the image obtained as a result of shooting. Therefore, in this case, the image is reduced at the reduction ratio in the standard state described above to detect a motion vector. On the other hand, when the shutter speed exceeds "1 / focal length" seconds, the reduction ratio is increased, that is, the image size is increased and the motion vector is detected.

  Thus, the reduction ratio determination circuit 106 determines the image reduction ratio according to the shutter speed under the control of the microcomputer 112, and sends the reduction ratio to the image reduction circuits 107 and 108.

  Referring again to FIG. 2, each of the image reduction circuits 107 and 108 reduces the image according to the reduction ratio determined by the reduction ratio determination circuit 106 (step S203). Here, an original image and a reference image used to detect a motion vector are selected from the plurality of frame images stored in the memory 104 at the same size in step S201, and input to the image reduction circuits 107 and 108. Then, the image reduction circuits 107 and 108 respectively reduce the original image and the reference image based on the image reduction ratio determined by the reduction ratio determination circuit 106.

  As for the method of image reduction, for example, if the reduction ratio is 1/8, the average value of the pixel values in the 8 × 8 rectangular area is calculated to use the pixel average method for reducing the image. It is also good. Alternatively, the image may be reduced using a so-called reduction filter that weights pixel values located in the center.

  Subsequently, the motion vector detection circuit 109 detects a motion vector in the two frame images (images subjected to reduction processing) input from the image reduction circuits 107 and 108 (step S204). For example, the motion vector detection circuit 109 detects a motion vector using template matching.

  FIG. 9 is a diagram for explaining template matching performed by the motion vector detection circuit 109 shown in FIG. FIG. 9A is a view showing an example of an original image, and FIG. 9B is a view showing an example of a reference image. Note that FIG. 9 shows an image in which a frame image of the same size stored in the memory 104 is subjected to reduction processing.

  The motion vector detection circuit 109 arranges the template block 901 at an arbitrary position in the original image. Then, the motion vector detection circuit 109 performs template matching on the reference image by the template block 901, and calculates a correlation value in each block region of the reference image. At this time, if the correlation value is calculated for all block areas of the reference image, the amount of calculation becomes enormous. Therefore, the motion vector detection circuit 109 sets a rectangular area for calculating a correlation value in the reference image as the search range 902. Here, although there is no particular limitation on the position and size of the search range 902, if the block range corresponding to the moving destination of the template block 901 is not included in the search range 902, a motion vector can be detected correctly. Can not.

  In the illustrated example, a sum of absolute differences (hereinafter referred to as SAD) method is used as a method of calculating the correlation value.

  When obtaining the correlation value S_SAD, the motion vector detection circuit 109 uses the following equation (1).

  In equation (1), f (i, j) represents the luminance value at the coordinates (i, j) of the template block 901, and g (i, j) represents the block area 903 targeted for correlation value calculation in the search range 902. Indicates the luminance value of Then, in the SAD method, the absolute value of the difference between the luminance value f (i, j) of the template block 901 and the luminance value g (i, j) of the block area 903 is determined, and the sum is calculated. Get the value S_SAD.

  Therefore, the smaller the correlation value S_SAD is, the smaller the difference in luminance value between the template block 901 and the block area 903 is. That is, the textures of the template block 901 and the block area 903 are similar.

  Here, although the correlation value is obtained by the SAD method, for example, a sum of squared difference (SSD) or a normalized cross correlation (NCC) may be used. However, when using a method other than SAD, there are cases where the degree of similarity is higher as the correlation value is smaller and the degree of similarity is higher as the correlation value is larger. Therefore, depending on the relationship between the correlation value and the degree of similarity, You also need to change the processing of.

  The motion vector detection circuit 109 moves the block area 903 in the entire area of the search range 902, and calculates the template block 901 and the correlation value in the entire area of the search range 902. Then, the motion vector detection circuit 109 determines a position (that is, a block area) where the correlation value is the smallest, and to which position in the reference image the template block on the original image has moved, that is, reference to the original image Detect motion vectors between images.

  Thus, the motion vector detection circuit 109 detects motion vectors for a plurality of regions between frame images. That is, a template block is set at another position in the original image, and motion vectors are similarly detected to obtain a motion vector group. Then, the motion vector detection circuit 109 sends the motion vector group to the geometric deformation parameter estimation circuit 110.

  Subsequently, the geometric deformation parameter estimation circuit 110 estimates geometric deformation parameters between frame images according to the motion vector group (step S205). Here, as an example of a model of geometric deformation, the case where the homography matrix is used as the image deformation amount will be described.

  Now, it is assumed that a certain point a on one frame (image) is represented by equation (2), and the point a moves to a point a ′ shown in equation (3) in the next frame.

  In the equations (2) and (3), the suffix T indicates that it is a transposed matrix.

  The correspondence between the point a shown in the equation (2) and the point a 'shown in the equation (3) is expressed by the equation (4) using the homography matrix H.

  The homography matrix H is a matrix equation showing the amount of deformation due to translation, rotation, scaling, shear, and tilt between images, and is expressed by the following equation (5).

  Each element of the homography matrix H is calculated by performing statistical processing such as the least squares method according to the motion vector group obtained in the processing of step S 204, that is, the correspondence of representative points between frame images. Ru. However, since the motion vector detected from the frame image subjected to the reduction processing is the motion vector in the reduced image, when estimating the geometric deformation parameter, it is necessary to convert the motion vector into one having the same size. is there.

  The homography matrix H determined as described above indicates the amount of image deformation caused by camera shake. Therefore, when correcting the blurring of the image, it is necessary to convert the homography matrix H so as to obtain an image deformation amount that cancels the deformation due to the blurring. That is, by converting the homography matrix H into the inverse matrix H, the correspondence between the point a 'and the point a is expressed by the following equation (6).

  The equation (6) makes it possible to return the point a 'after the occurrence of blurring to the same coordinates as the point a before the occurrence of blurring.

  In the illustrated example, the homography matrix is used as a model representing the amount of blurring between frame images, but another model such as a Helmert matrix or an affine matrix may be used, for example. Furthermore, when the main subject is set as the image stabilization target, it is sufficient to obtain only the translational shake correction amount. In this case, since the average value of the motion vector group may be obtained, statistical estimation is performed. It is unnecessary to reduce the amount of calculation.

  Next, the microcomputer 112 gives the geometric deformation circuit 111 the geometric deformation parameters obtained as described above. The geometric deformation circuit 111 performs geometric conversion processing on the frame image stored in the memory 104 using the geometric deformation parameter to perform image stabilization processing (step S206). Then, the geometric deformation circuit 111 stores the image subjected to the image stabilization processing in a storage device (not shown) and displays the image on a display device (not shown). Thereafter, the microcomputer 112 ends the photographing process.

  As described above, in the first embodiment of the present invention, the reduction ratio of the image used to detect the motion vector is changed according to the shutter speed. By this, it is possible to accurately detect the motion vector regardless of the shutter speed, and correct the blurring in the image.

Second Embodiment
Subsequently, an example of a camera according to a second embodiment of the present invention will be described.

  FIG. 10 is a block diagram showing the configuration of an example of a camera according to the second embodiment of the present invention. In FIG. 10, the same components as those in the camera shown in FIG.

  The illustrated camera includes an image resizing circuit 1001 instead of the image reducing circuit 108.

  FIG. 11 is a flowchart for explaining an example of the photographing process performed by the camera shown in FIG. In the flowchart shown in the figure, the same steps as those in the flowchart shown in FIG.

  When photographing is started, under the control of the microcomputer 112, the camera signal processing circuit 103 records the frame image in the memory 104 and sends the frame image to the image reduction circuit 107 (step S1101). In step S202, the reduction ratio determination circuit 106 determines the image reduction ratio as described above, and sets the reduction ratio in the image reduction circuit 107 and the image resizing circuit 1001.

  Subsequently, the image reduction circuit 107 reduces the frame image sent from the camera signal processing circuit 103 based on the reduction ratio set by the reduction ratio setting circuit 106 (step S1103). Then, the image reduction circuit 107 stores the reduced frame image in the memory 104.

  The image resizing circuit 1001 resizes the reduced frame image stored in the memory 104 (step S1104).

  By the way, the reduced frame image stored in the memory 104 is reduced according to the reduction ratio determined by the reduction ratio determining circuit 106 at the time of shooting. However, when the reduction ratio is changed according to the temporal change of the photographed scene, the reduced frame image already stored in the memory 104 and the frame image on which the reduction processing has been performed after the reduction ratio is changed. There is a difference in image size among them. Then, if the image size is different between the original image and the reference image, the motion vector can not be detected. Therefore, the image resizing circuit 1001 resizes the reduced image stored in the memory 104 in order to match the size of both the original image and the reference image.

  For example, it is assumed that the reduction ratio at the shooting time (that is, input time) for a certain frame image is 1/8. Then, when the reduction ratio is changed to 1⁄4 according to the temporal change of the photographed scene, the image resizing circuit 1001 doubles the 1⁄8 reduced frame image stored in the memory 104 to process the original. It is sent to the motion vector detection circuit 109 as an image. On the other hand, the frame image of 1⁄4 reduction is sent from the image reduction circuit 107 to the motion vector detection circuit 109 as a reference image and stored in the memory 104.

  As a result, since both the original image and the reference image have a 1⁄4 reduced size, the motion vector detection circuit 109 can detect a motion vector using the original image and the reference image of the same image size.

  The resizing method performed by the image resizing circuit 1001 may be any method, for example, bilinear or bicubic interpolation processing may be used.

  Thereafter, after the processing of steps S204 to S207 described in FIG. 2 is performed, the microcomputer 112 ends the photographing processing.

  As described above, in the second embodiment of the present invention, the frame image is reduced and stored in the memory 104 at the reduction rate at the time of shooting. Then, the reduced frame image stored in the memory 104 is resized according to the change in the reduction ratio. In the first embodiment described above, only the frame image of the same size is stored in the memory 104. Therefore, the image reduction circuit reads the frame image of the same size from the memory 104 when there is a change in the reduction ratio. Image reduction processing will be performed. Therefore, the bandwidth for transmission becomes large. On the other hand, in the second embodiment, the frame image is reduced at the reduction rate at the time of shooting and stored in the memory 104, and the reduced frame image stored in the memory 104 is resized according to the change in reduction rate. As a result, the bandwidth for transmission can be reduced.

  As apparent from the above description, in the examples shown in FIGS. 1 and 10, the image reduction circuits 107 and 108 function as reduction means, and the microcomputer 112 and the reduction ratio determination circuit 106 function as determination means. The motion vector detection circuit 109, the geometric deformation parameter estimation circuit 110, the geometric deformation circuit 111, and the image resizing circuit 1001 respectively function as a detection unit, a calculation unit, a correction unit, and a resizing unit.

  Further, in the above embodiment, an example in which the image stabilization processing is performed inside the imaging apparatus has been described, but the present invention is not limited to this. The imaging device may associate and output a shutter speed corresponding to a captured image, and may perform image reduction and motion vector detection with a personal computer or the like that has received these. That is, any image processing apparatus having a function of reducing a plurality of images according to the shutter speed and detecting a motion vector using these may be used.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, The various form of the range which does not deviate from the summary of this invention is also included in this invention .

  For example, the control method may be executed by the image processing apparatus as the control method of the above-described embodiment. Further, a program having the functions of the above-described embodiments may be used as a control program to cause a computer provided with the image processing apparatus to execute the control program. The control program is recorded, for example, on a computer readable recording medium.

Other Embodiments
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

102 image sensor 103 camera signal processing circuit 104 memory 106 reduction ratio determination circuit 107, 108 image reduction circuit 109 motion vector detection circuit 110 geometric deformation parameter estimation circuit 111 geometric deformation circuit 112 microcomputer 1001 image resize circuit

Claims (7)

Reducing means for reducing the original image and the reference image by using one image as an original image and using an image obtained as a result of shooting after the original image as a reference image;
A determination unit that determines a reduction ratio when the reduction processing is performed by the reduction device according to a shutter speed at the time of capturing the original image and the reference image;
Detection means for detecting a motion vector between the original image and the reference image based on the original image and the reference image subjected to the reduction processing;
I have a,
The image processing apparatus , wherein the determination unit determines the reduction ratio so that the image size increases as the shutter speed decreases . Memory means for storing the original image and the reference image;
The image processing apparatus according to claim 1, wherein the reduction unit performs the reduction process by reading the original image and the reference image from the memory unit. Memory means for storing the original image and the reference image subjected to the reduction process;
The apparatus further comprises resizing means for performing resizing processing to equalize the reduction rates of the original image and the reference image subjected to the reduction processing when the reduction ratio according to the shutter speed changes during the photographing. The image processing apparatus according to claim 1, wherein Calculating means for determining a blur between the original image and the reference image as a geometric deformation parameter based on the motion vector;
4. The image according to any one of claims 1 to 3 , further comprising: correction means for geometrically deforming an image obtained as a result of shooting based on the geometric deformation parameter to correct blurring of the image. Processing unit. Imaging means for imaging the original image and the reference image;
An image pickup apparatus comprising the image processing apparatus according to any one of claims 1 to 4 . Reducing an original image and the reference image by using one image as an original image and using an image obtained as a result of shooting after the original image as a reference image;
A determination step of determining a reduction ratio when the reduction processing is performed in the reduction step in accordance with a shutter speed at the time of photographing the original image and the reference image;
Detecting a motion vector between the original image and the reference image based on the original image and the reference image subjected to the reduction processing;
I have a,
The control method of the image processing apparatus, wherein in the determination step, the reduction ratio is determined such that the image size becomes larger as the shutter speed becomes slower . A control program used in the image processing apparatus,
A computer provided in the image processing apparatus;
Reducing an original image and the reference image by using one image as an original image and using an image obtained as a result of shooting after the original image as a reference image;
A determination step of determining a reduction ratio when the reduction processing is performed in the reduction step in accordance with a shutter speed at the time of photographing the original image and the reference image;
Detecting a motion vector between the original image and the reference image based on the original image and the reference image subjected to the reduction processing;
It is those for the execution,
The control program , wherein in the determination step, the reduction ratio is determined so that the image size increases as the shutter speed decreases .