CN101562704B

CN101562704B - Image processing apparatus and image processing method

Info

Publication number: CN101562704B
Application number: CN2009101344528A
Authority: CN
Inventors: 仓田徹
Original assignee: Sony Corp
Current assignee: Sony Corp
Priority date: 2008-04-15
Filing date: 2009-04-15
Publication date: 2012-07-18
Anticipated expiration: 2029-04-15
Also published as: CN101562704A; EP2111046A2; JP4544334B2; US8446957B2; US20090257498A1; JP2009258868A; EP2111046A3

Abstract

The present invention discloses an image processing apparatus and an image processing method. The image processing apparatus includes: motion vector detection section configured to detect a motion vector of each of a plurality of blocks of a predetermined size set in an image and formed from a plurality of pixels; and global motion calculation section configured to carry out convergence mathematical operation, from the motion vectors of the blocks detected by the motion vector detection section, using extended affine transformation in which at least one of affine parameters is represented by afunction of a variable regarding a displacement axis of the image to calculate a global motion representative of deformation applied to the entire image.

Description

Image processing apparatus and image processing method

Quoting of related application

The present invention includes on the April 15th, 2008 of disclosed theme in the japanese priority patent application JP 2008-105248 that Japan Patent office submits to, and its full content is hereby expressly incorporated by reference.

Technical field

The present invention relates to be used for the image processing apparatus and the image processing method of global motion of the distortion of the entire image of represents between two width of cloth pictures (screen image).

Background technology

The block-matching technique of confirming the motion vector between two width of cloth pictures according to image information itself is a kind of technology with very long history.

Block-matching technique is a kind of method that is used to calculate the motion vector between two width of cloth pictures, and this two width of cloth picture comprises: reference picture, and it is for being paid close attention to picture; And raw frames (hereinafter, being called target picture), the motion of reference picture comes from this raw frames.According to block-matching technique, calculate the correlation between reference picture and the target picture through piece (object block and reference block) with respect to the rectangular area of preliminary dimension, come calculating kinematical vector.

Block-matching technique comprises two kinds of situation, comprises target picture in time prior to the situation of reference picture, and reference picture is in time prior to another situation of target picture.Before a kind of instance of situation be the motion detection through MPEG (Motion Picture Experts Group) system, then a kind of instance of situation comes noise reduction for the stack of the picture frame through hereinafter described.

Notice that in this manual, the term picture is represented the formed image of view data by a frame or a field.Yet, for convenience, in the following description of this specification, suppose to form a width of cloth picture by a frame.Therefore, hereinafter, a width of cloth picture is known as frame.Therefore, reference picture is called reference frame hereinafter, and target picture is called target frame hereinafter.

Particularly to the image pickup target following of television camera and The Cloud Terrace (pan-tilt) detect, the mobile image code of mpeg system etc. has been developed the piece coupling method of detecting motion vector.In the nineties, promoted to comprise the no sensor camera jitter correction realized through image overlay, the extensive use of the noise reduction in the low-light level environment during image taking etc.

In addition, the method for detecting motion vector of piece coupling not only is applied to the image recognition application and camera shake correction is used, and is applied to the new application of image pattern as the double speed converting frame rate of automatic adjusting of the shutter speed of pick device and liquid crystal TV set.

Incidentally; Detection is arranged on every motion vector in a large amount of object block in the picture, that is, and and local motion vector; And use a large amount of local motion vectors that detect by this way to calculate global motion, said global motion representes to be applied to the distortion of the entire image between two width of cloth pictures.Global motion is often referred to the motion and the amount of exercise of the picture background of publishing picture.

As prior art, disclose in 2007-221631 number (hereinafter, being called patent document 1) at Japan Patent and to disclose a kind of aberration emendation method, this method is divided into a plurality of and in these pieces each confirmed a vector with a width of cloth picture.Then, determined by this way motion vector directly is used to calculate global motion.The technology of patent document 1 has been used as the camera shake correction technology that is mainly used in mobile picture, and up to several years ago, pixel quantity at that time is less.

For disclosed technology in patent document 1, can use the global motion of hardware size detection cheaply, and realization is used for the mobile picture of high picture quality and the good no transducer of still picture does not perhaps have gyroscope camera shake correction and noise reduction.

Yet, under the situation that a picture is divided into bigger piece (for example about 16), there is such problem, be difficult to follow the tracks of mobile image pickup object when in image, comprising when moving image pickup object.

Therefore, the if block number increases to for example about 64, then can use a large amount of local motion vectors about smaller image-region, therefore, mobile the mobile of image pickup object is reached to a certain degree.Yet, under the situation that adopts this structure just described, use the technology of patent document 1 can produce such problem, hardware size increases and mostly can not reach as the cost advantage of the technical advantage of patent document 1.

Meanwhile, the method as confirming global motion discloses the method that affine transformation (affine transformation) is applied to the local motion vector of a plurality of detections.

Figure 34 shows the general formula like the affine transformation of (expression formula 1).With reference to Figure 34, in (expression formula 1), v representes the horizontal component of the mobile vector of object block, and w representes the vertical component of the mobile vector of object block, and a, b, c, d, e and f represent affine parameter.In affine transformation commonly used, affine parameter a, b, c, d, e and f are fixed value.In addition, x and y represent the horizontal component and the vertical component of the centre coordinate of object block respectively.

Adopt the centre coordinate of each object block and handle the affine parameter of confirming, obtain and the corresponding motion vector of global motion through the convergence mathematical operation of global motion.Hereinafter, this motion vector of just having been mentioned is called the ideal movements vector.Such as the desirable vector of Figure 35 (expression formula 2) expression with observe (through piece mate detection) motion vector between the summation ε of error.

The proposition of deriving global motion is the affine parameter a to f that estimation minimizes the summation ε of above-mentioned error, and can solve through for example least square method.Figure 36,37 and 38 expression formula 3, expression formula 4 and expression formula 5 show the derivation process of affine parameter a to f and result that should the derivation process respectively.

When comparing the parameter of easy for calculation affine transformation by this way, its effect is also bigger.Because affine transformation not only can move, rotate and expand or shrink to the parallel of image, but also can offset most camera shake to distortion to a certain degree, that is, and the accurate correction of camera operation.

For example, disclose in 2005-321902 number (hereinafter, being called patent document 2) at Japan Patent and disclose aforesaid this affine transformation.

Summary of the invention

Yet the shortcoming of affine transformation is, can not be used for " oblique distortions "." oblique distortions " is the phenomenon that the rotational component by pitch axis of camera shake (be vertical direction axle) or yaw axis (axle of promptly vertical with vertical direction horizontal direction) causes.Because " oblique distortions " will become the trapezoidal shape shown in Figure 39 B over against the rectangle plane distortion of state, therefore, be also referred to as trapezoidal distortion or trapezoidal distortion.

Particularly; When camera CAM over against image pickup object OBJ; Thereby the optical axis L z of camera CAM vertically extends to as when the rectangle plane of the image pickup target OBJ shown in the downside among Figure 39 A, and the image through picking up of the rectangle plane of image pickup object OBJ directly is rendered as as at the rectangular shape shown in the upside among Figure 39 A.

On the other hand; For example, if the optical axis L z of camera CAM has carried out the pitch axis rotation (promptly in vertical plane, rotating) of angle θ, and not as shown in the downside of Figure 39 B over against image pickup object OBJ; Then the image through picking up of the rectangle plane of image pickup object OBJ is trapezoidal shape; Wherein, in response to as at the angle θ shown in the upside of Figure 39 B, change linearly along the left direction of image and the length of right direction (being horizontal direction).

Should note; Although do not illustrate,, but carried out the yaw axis rotation (promptly in horizontal plane, rotating) of angle θ if the optical axis L z of camera CAM is not over against image pickup object OBJ; Then the image through picking up of the rectangle plane of image pickup object OBJ is trapezoidal shape; Wherein, in response to angle θ, change linearly along the last side direction of image and lower side length to the vertical direction of image (promptly along).

Be used for to return to owing to the image that the aforesaid this variant of camera shake is picked up that to be in over against the treatment of picture of the original-shape of state be that " oblique distortions " proofreaies and correct or keystone.Yet, can not be with this correction of the parametric representation of above-mentioned affine transformation.Particularly; If look like to return to this " oblique distortions " of the image of rectangular shape proofreaies and correct for the ladder diagram of carrying out this " oblique distortions " that will be shown in Figure 39 B; Then in horizontal extension image bottom, should level shrink upper image along vertical axis or y axle.Yet in the parameter of affine transformation, the parameter of participating in horizontal extension or contraction is a, and because parameter a is a fixed value, does so affine transformation can not be used for above-mentioned correction.

For fear of this problem of just having described, proposed use rectangular projection conversion and replaced affine transformation.Represent the rectangular projection conversion by expression formula shown in Figure 40 6, and can represent in the three dimensions projective transformation from arbitrary plane to another plane.

Yet, be difficult to the rectangular projection conversion is applied to this situation: as This document assumes that, stipulate a global motion by a plurality of vectors.This is because be difficult to the shape use least square method according to the expression formula 6 of Figure 40, in addition, under some other technological situation of use, also needs a large amount of mathematical operations.

A realistic problem is, because actual image pickup object is a three dimensional object, so comprise the different a plurality of image pickup parts of distance each other through the local motion vector of observation.In this case, the application of rectangular projection conversion is nothing but simple application.Therefore, below seem rational judgement: if considered to obtain the very high cost of general solution needs of rectangular projection conversion, then the rectangular projection conversion is not suitable as and is used for the countermeasure that cost reduces.

In order to use simpler rectangular projection conversion, can adopt with respect to 6 vectors and confirm the way of separating.This is because owing to must confirm 12 parameters, if replaced the coordinate of six vectors, then generation has 6 * 2 simple equations of six unknown numbers, and can relatively easily derive through 6 * 6 inverse of a matrix matrix computations and separate.Therefore, only need from a plurality of Local Vector, suitably to select six vectors.

Yet; If consider the situation that moves image pickup object in the image that picks up, including; Then, have high-precision those vectors so be difficult to from a plurality of local motion vectors, only extract owing to can not expect that each vector in the Local Vector all has sizable precision.Therefore, be difficult to from a large amount of local motion vectors, suitably select six vectors.

Therefore, even image has " oblique distortions " distortion, we also expect to provide the technology that can successfully detect global motion.

According to embodiments of the invention, a kind of image processing apparatus is provided, comprising: motion vector detection section is configured to detect each the motion vector among a plurality of of preliminary dimension that is arranged in the image and formed by a plurality of pixels; And global motion calculating part; Be configured to utilize at least one affine parameter wherein by affine transformation about the expansion of the function representation of the variable of the offset axis of image; Employing is carried out the convergence mathematical operation by the motion vector of the piece that motion vector detection section detected, and puts on the global motion of the distortion of entire image with represents.

In above-mentioned image processing apparatus, replacing all affine parameters is the affine transformation of fixed value, and at least one in the use affine transformation parameter is the affine transformation about the expansion of the function of the variable of the offset axis of image.

For example; If we for example manage to carry out " oblique distortions " of the image that will this " oblique distortions " ladder diagram shown in Figure 39 B looks like to return to rectangular shape and proofread and correct; Then should be when flatly expanding the bottom graph picture, along the top of vertical axis or y axle horizontal ground contractible graph picture.According to the affine parameter in the expression formula 1 of affine transformation shown in Figure 34, the parameter of participating in horizontal extension or contraction is a.Therefore, will then can calculate this global motion of " oblique distortions " distortion of also having carried out the image shown in Figure 39 B with respect to the function of the variable y of vertical axis or y axle affine transformation if for example used as the expansion of affine parameter a.

Since the expansion affine transformation comprise increase transformation parameter quantity and be variable; So exist to possibility, thereby might restrain mathematical operation and can not get rid of error vector such as the erroneous motion vectors component that moves image pickup object or noise.

Therefore; Preferably; Image processing apparatus is configured to, in the convergence mathematical operation of the initial stage of global motion calculating part, through carried out the affine transformation commonly used of the affine parameter that has used fixed coefficient by the motion vector of the piece that motion vector detection section detected; And after the convergence mathematical operation in the early stage, brought into use the convergence mathematical operation of the affine transformation of expansion.

In image processing apparatus, in the convergence mathematical operation of the initial stage of global motion calculating part, carry out the affine transformation commonly used of the affine parameter that has used fixed coefficient.Then, from motion vector components, get rid of through motion vector detection section after, carry out the convergence mathematical operation of the affine transformation of having used expansion such as the erroneous motion vectors component that moves image pickup object or noise.

According to another embodiment of the present invention, a kind of image processing method is provided, has comprised: the motion vector detection step, detect each the motion vector among a plurality of of preliminary dimension be arranged in the image and formed by a plurality of pixels; And global motion calculation procedure; Utilize at least one affine parameter wherein by affine transformation about the expansion of the function representation of the variable of the offset axis of image; The motion vector that is employed in the piece that is detected in the motion vector detection step is carried out the convergence mathematical operation, puts on the global motion of the distortion of entire image with represents.

In a word, for image processing apparatus, owing to used at least one affine parameter wherein by affine transformation, so can calculate the global motion of " oblique distortions " distortion of also having considered image about the expansion of the function representation of the variable of the offset axis of image.

Combine the specific descriptions and the accompanying claims of accompanying drawing from hereinafter, will more be expressly understood above-mentioned and further feature and advantage of the present invention, wherein, same reference marker is represented identical parts or element.

Description of drawings

Fig. 1 is the block diagram of instance that the structure of the image pick-up device of having used image processing apparatus according to a preferred embodiment of the invention is shown;

Fig. 2 A, Fig. 2 B and Fig. 3 to Figure 10 are the sketch mapes that illustrates through the piece matching treatment of image processing apparatus;

Figure 11 is the block diagram that illustrates through the piece matching treatment of image processing apparatus;

Figure 12 illustrates to have used the sketch map of the instance of the image that picks up of image processing method according to an embodiment of the invention;

Figure 13 is the sketch map that uses the local motion vector that image processing method detected in the dependency relation that is illustrated in the image that picks up of Figure 12;

Figure 14 is the graphical diagram that is illustrated in the SAD table that uses in the image processing method;

Figure 15 is the graphical diagram that is illustrated in the SAD table of the Figure 14 that is obtained on the one dimension axle;

Figure 16 is illustrated in the dependency relation of the image that picks up of Figure 12 the diagrammatic sketch of confirming to have the local motion vector of high reliability through image processing method;

Figure 17 is the block diagram of embodiment of structure that the global motion vector computing block of image processing apparatus is shown;

Figure 18 is the graphical diagram that the operation of global motion vector calculating part is shown;

Figure 19 is the flow chart that the instance of operation is handled in detection that the local motion vector computing block through image processing apparatus is shown;

Figure 20 is the sketch map that common affine transformation is shown;

Figure 21 to Figure 27 illustrates by what image processing apparatus was carried out to calculate the diagrammatic sketch of instance of the processing of global motion with local motion vector;

Figure 28 and Figure 29 illustrate by what image processing apparatus was carried out to calculate the flow chart of instance of the processing of global motion with local motion vector;

Figure 30 is through the diagrammatic sketch of image processing apparatus from the motion vector of definite module unit through the global motion that calculates in the dependency relation that is illustrated on the image;

Figure 31 and Figure 32 are the graphical diagram that another instance of the processing of confirming the use local motion vector that graphic processing method detected is shown;

Figure 33 is the graphical diagram that the another instance of the processing of confirming the use local motion vector that image processing method detected is shown;

Figure 34 to Figure 38 is the diagrammatic sketch that is illustrated in the expression formula of being confirmed to use in the instance of processing of global motion by local motion vector;

Figure 39 A and Figure 39 B are the graphical diagram that the problem of using affine transformation commonly used, being confirmed by local motion vector to exist under the situation of global motion is shown in the prior art; And

Figure 40 is the diagrammatic sketch that is illustrated in the expression formula of using the rectangular projection conversion, being confirmed by local motion vector to use in the processing of global motion;

Embodiment

Hereinafter, described the present invention in detail, wherein, will be applied to image pick-up device according to image processing apparatus of the present invention and image processing method in conjunction with its preferred embodiment.

[first embodiment]

Image pick-up device according to first embodiment is configured to usually, will be by a plurality of still images location and the stack each other that image pick-up device or imager picked up, to reduce the noise of image.

In the environment of the hand-held image pick-up device captured image of user, camera shake possibly take place.For example pick up through continuous shooting under the situation of static images at the image pick-up device that utilizes the user to hand by this way, first static images be used as target frame and second and static images subsequently be used as reference frame.

Parallelly move through comprising, rotation and expansion or the conversion process of shrinking be applied to whole reference frame and can reference frame be positioned under the situation of target frame position, the conversion process that is applied to whole reference frame is a global motion.Moving and amount of movement of global motion ordinary representation image background.

Camera shake is represented with respect to target frame a certain amount of global motion to have taken place, said a certain amount of be reference frame with respect to parallelly move target frame comprising of carrying out, rotation and expansion or the image transform of shrinking.

In the description of present embodiment, first width of cloth in the static images that picks up is used as benchmark, that is, and and as the target frame that is used to simplify description.Yet this is optional.Can use arbitrarily n open static images as target frame and use arbitrarily m (n ≠ m) open static images as a reference frame define global motion again.Perhaps, can be to two different on the time in moving picture two field picture definition global motions.Otherwise global motion is handled the interior whole effective picture frame of a frame of the image that can shall not be applied to hangs oneself picks up, and only is applied to the parts of images of a frame.

If confirmed global motion, then can be according to the motion vector that is applied to the motion of the whole image between target frame and reference frame through the global motion represents of confirming.Hereinafter, be called global motion vector like this motion vector of just having mentioned.Can be with respect to every definite global motion vector in a plurality of object block that are arranged on the picture.

Then, if can accurately confirm global motion vector based on global motion, correcting camera shake aptly then.Perhaps, can in camera jitter correction global motion vector, a plurality of two field pictures be superposeed each other.

In the following description; Use motion vector detection and motion compensation to superpose a plurality of images to reduce noise (wherein; Use two width of cloth images of motion vector location different frame) be known as noise reduction, and be known as the noise reduction image through the image that such noise reduction reduces its noise.

In the present embodiment, detect motion vector through above-mentioned block matching method.In addition, in the present embodiment, a width of cloth picture is divided into a plurality of, and to come with the piece through block matching method be that unit detects motion vector (hereinafter, this motion vector is called local motion vector).In addition, the corrected value in the use block matching method detects the reliability index of local motion vector with following this mode.Then, only according to this in the local motion vector that detects have a high reliability those calculate global motion, and the global motion vector of each piece (for object block hereinafter described) in detecting a plurality of according to the global motion that calculates.

[main points of piece coupling]

Fig. 2 A to Fig. 7 shows the main points of block matching method.Here, in described block matching method, in the target frame shown in Fig. 2 A 100, the piece (being object block 102) of supposing the rectangular area of preliminary dimension comprises a plurality of pixels and the many lines (line) on the vertical direction on the horizontal direction.

In piece coupling, search out the piece that has with the high correlation of object block 102 from reference frame 101 inside.Reference block 103 shown in the inner Fig. 2 B that detects as the piece that has with the high correlation of object block 102 of reference frame 101 hereinafter is called motion compensation block.In addition, the displacement between object block 102 and motion compensation block 103 (and object block 102 has high correlation) hereinafter is called motion vector (with reference to the reference marker 104 of figure 2B).

Corresponding to the motion vector 104 of the displacement between object block 102 and motion compensation block 103 (comprising displacement and direction of displacement) corresponding to the projected image piece 109 of object block 102 such as the position of center and motion compensation block 103 such as the displacement between the position of center; Wherein, the projected image piece 109 of hypothetical target piece 102 is in the position of the reference frame 101 identical with the position of the object block 102 of target frame 100.Therefore, motion vector 104 has the durection component of displacement and displacement.

The main points of piece matching treatment are described below.With reference to figure 3; The projected image piece 109 of the object block 102 of hypothetical target frame 100 in certain position of reference frame 101 (shown in the dotted line of Fig. 3; This position is identical with the position of the object block 102 of target frame 100), and the centre coordinate of the projected image piece 109 of object block 102 confirmed as the initial point 105 that is used for motion detection.Then, suppose that motion vector 104 is present in the particular range of the initial point 105 that is used for motion detection, and the preset range that concentrates on the initial point 105 that is used for motion detection is set to the indicated hunting zone 106 of chain-dotted line of Fig. 3.

Then, the piece (being reference block 108) with object block 102 same sizes is set on the reference picture.Then, in such as hunting zone 106, be the position that unit moves reference block 108 with a pixel or a plurality of pixels along horizontal direction and vertical direction.Therefore, a plurality of reference blocks 108 are set in the hunting zone 106.

Here; In hunting zone 106, move reference block 108; This hunting zone is represented: owing to be used for the initial point 105 of motion detection is the center of object block; So in hunting zone 106, move the center of reference block 108, and the pixel of formation reference block 108 possibly given prominence to from hunting zone 106.

Then, for each reference block 108 that is arranged in the hunting zone 106, expression reference block 108 and the displacement of object block 102 and the vector (being reference vector 107) (with reference to figure 3) of direction of displacement are set.Then, estimate correlation between the picture material of the picture material of the reference block 108 of the indicated position of each this reference vector 107 and object block 102.

With reference to figure 4, (Vx, Vy), wherein, Vx is the displacement of the reference block 108 of (being on the directions X) in the horizontal direction, and Vy is the displacement of the reference block 108 of (being on the Y direction) in vertical direction can reference vector 107 to be expressed as vector.If the position coordinates (such as center position coordinates) of position coordinates of reference block 108 (such as center position coordinates) and object block 102 is identical, then reference vector 107 is expressed as vector (0,0).

For example, if reference block 108 leaves the position apart from displacement that there is a pixel position of object block 102 on directions X, then reference vector 107 is represented vectors (1,0).Meanwhile, if reference block 108 in the position that two pixels are arranged apart from displacement and on the Y direction that three pixels are arranged from the position of object block 102 on the directions X apart from displacement, then reference vector 107 is vector (3,2).

In brief, as shown in Figure 5, reference vector 107 is illustrated between each reference block 108 that corresponds to each other and the object block 102 as the displacement that comprises the vector of displacement and direction of displacement.It should be noted that in Fig. 5, represent the position of object block 102 and reference block 108 respectively by the center of piece.

Reference block 108 moves in hunting zone 106, and in this case, move in hunting zone 106 center of reference block 108.Because reference block 108 is included in a plurality of pixels on aforesaid horizontal direction and the vertical direction; So provide object block 102 with as the maximum magnitude that moves of the reference block 108 of piece matching treatment object as matching treatment scope 110, this matching treatment scope is wide such as hunting zone shown in Figure 5 106.

Then, has the piece of maximum correlation and the position of the reference block 108 that detects is detected as the position of object block 102 on reference frame 101 of target frame 100 as picture material, promptly as the position after moving with object block 102.Then, will confirm as above-mentioned motion compensation block 103 through the reference block that detects.Then, detect at the displacement between the position of the position of the motion compensation block 103 that detects and object block 102 as the motion vector 104 that comprises the durection component shown in Fig. 2 B.

Basically, use the respective pixel value of object block 102 and reference block 108 to come represents object block 102 and the correlation of the degree of correlation between the mobile reference block 108 hunting zone 106 in.The calculating of correlation can be used and comprise the various computational methods that adopt root mean square method.

For example; A value among the correlation that when calculating kinematical vector, generally uses is; For example, with respect to the summation of the absolute value of the difference between the brightness value of respective pixel in the brightness value of each pixel in the object block 102 of all pixels in the piece shown in Figure 6 and the hunting zone 106.The summation of the absolute value of this difference is called poor absolute value sum, and hereinafter is called SAD (absolute difference sum) value.

Be used as at sad value under the situation of correlation, when sad value reduced, correlation increased.Therefore, in the reference block 108 that in hunting zone 106, moves, the reference block 108 that is in the minimum position of sad value is to have the highest coherent reference piece of high correlation.Detect this highest coherent reference piece as motion compensation block 103, and the position of detecting object block 102 to through the displacement of the position of the motion compensation block 103 that detects as motion vector.

As stated, in piece coupling, through displacement as each in a plurality of reference blocks 108 that are arranged in the hunting zone 106 of the position of the reference vector 107 expression object block 102 of the amount that comprises durection component.The reference vector 107 of each reference block 108 has on reference frame 101 and the corresponding value in the position of reference block 108.As stated, in the piece coupling, the reference vector that detects reference block 108 is as motion vector 104, and wherein, the sad value of this reference block is a minimum as correlation.

Therefore, in piece coupling, at first confirm to be arranged in a plurality of reference blocks 108 in the hunting zone 106 each with object block 102 between sad value (hereinafter, this sad value is called the sad value of reference block 108, to simplify description).

Then, the sad value of confirming by this way is stored in the memory with the corresponding relation with the corresponding reference vector 107 in the position of reference block 108.Then, minimum one reference block 108 in the sad value with all reference blocks 108 of detection of stored in memory is to detect motion vector 104.It should be noted that in the following description hereinafter, the reference vector 107 that is called reference block 108 with the corresponding reference vector 107 in the position of reference block 108 is described to simplify.

Reference vector 107 correlations (in above-mentioned instance, being sad value) corresponding, that be arranged on the position of a plurality of reference blocks 108 in the hunting zone 106 with corresponding to the position of reference block 108 of storage are known as the correlation table.In described instance, owing to be used as correlation as the sad value of poor absolute value sum, the correlation table is called the table or the SAD table of poor absolute value sum.

It is expressed as the SAD table TBL of Fig. 7.With reference to figure 7, shown in SAD table TBL in, the correlation of each reference block 108 (in said instance, being sad value) is called the correlation table element.In the instance of Fig. 7, be the sad value when reference vector is vector (0,0) through reference number 111 indicated sad values.In addition, in the instance of Fig. 7, when reference vector is vector (3,2), because the minimum of sad value is " 7 ", so the motion vector 104 that warp is confirmed is vector (3,2).

The position that it should be noted that object block 102 and reference block 108 is meant any particular location such as the center of piece.In addition, reference vector 107 is illustrated in the displacement (comprising direction) between the position of position and reference block 108 of projected image piece 109 of object block 102 of reference frame 101.

With each reference block 108 corresponding reference vector 107 be on reference frame 101 with the displacement of the corresponding projected image pieces 109 of object block 102 to reference block 108.Therefore, if specified the position of reference block 108, then also specified value with this corresponding reference vector in position.Thereby, specified the address of the correlation table element of the reference block in the memory of SAD table TBL, specify the corresponding reference vector then.

It should be noted that and to calculate this sad value concurrently with respect to two or more object block.If the object block number that is processed simultaneously increases, then processing speed increases.Yet, increase owing to be used to calculate the scale of the hardware of sad value, so need balance to increase processing speed and increase the relation between the circuit scale.

[piece coupling embodiment]

Above-described coupling relates to the calculating of the motion vector of an object block.Because the ratio that object block is occupied in target frame is lower usually, so be difficult to usually this motion vector is confirmed as global motion vector like this.

In the present embodiment, as shown in Figure 8, target frame 100 is divided into a plurality of object block 102 such as the size of 64 pixels * 64 lines, and each motion vector or the local motion vector 104B in definite object block.At this moment, in the present embodiment, the index of the reliability of while each local motion vector of confirming of represents.

Then, based on the reliability index of local motion vector, only extract those local motion vectors of confirming about target frame with high reliability.Then, calculate global motion and only uses the local motion vector calculating with high reliability of being extracted with the global motion vector of piece as unit.Then, what calculate be that the global motion vector of unit is used for reference frame with the piece is that unit is positioned to target frame with the piece, then, stack by this way two frames of location to generate the image of noise reduction.

Then, image pick-up device with high-speed picking-up a plurality of still images as shown in Figure 9, and confirms that the first static images captured image is as target frame 100 when the static images image pickup.Then, image pick-up device confirm to comprise second with the static images captured image of the predetermined quantity of follow-up static images captured image frame 101 and carry out stack as a reference.Then, the image of image pick-up device record stack gained is as the static images captured image.

Particularly, if the image pickup person depresses the shutter release button of image pick-up device, then with the still image of high-speed picking-up predetermined quantity.Then, on the still image that at first picks up (frame), a plurality of still images (frame) that stack and meter record were picked up in the time after a while.

Should note; Although do not describe in the present embodiment; But when mobile picture picks up, confirm the image of the image of current present frame from image pick-up element output, and the image of the confirming former frame in the past image of frame 101 as a reference as target frame 100.In other words, when moving image picks up, in order to carry out the noise reduction of current frame image, the image of the frame before stack present frame on the present frame.

[instance of the hardware configuration of image pick-up device]

Fig. 1 shows the instance of conduct according to the image pick-up device of the image processing apparatus of the embodiment of the invention.

With reference to Fig. 1; Shown image pickup device comprises the central processing unit (CPU) 1 that is connected to system bus 2, and comprises: image pickup signal treatment system 10, user operate input unit 3, are connected to image storage unit 4 and the record and the regenerating unit unit 5 of system bus 2.Although it should be noted that among Fig. 1 not illustrate, CPU 1 comprises that storage is used to carry out the ROM (read-only memory) of the program of various software processes, is used for the RAM (random asccess memory) of service area etc.

In response to image pickup calculating beginning operation of operating input unit 3 through the user, the image pickup signal treatment system 10 of the image pick-up device of Fig. 1 is carried out the recording processing of the captured image data that are described below.In addition, begin operation, the Regeneration Treatment of the captured image data of image pickup signal treatment system 10 executive loggings on the recording medium of record and regenerating unit unit 5 in response to the regeneration of picking up document image of operating input unit 3 through the user.

In image pickup signal treatment system 10, via the unshowned camera optics system that comprises image pickup lens 10L receive from the incident light irradiation of image pickup object on image pick-up element 11, with captured image.In the present embodiment, image pick-up element 11 is made up of CCD (charge coupled device) imager.It should be noted that image pick-up element 11 also can be made up of CMOS (complementary metal oxide semiconductors (CMOS)) imager.

In image pick-up device,, then will become picking up image signal through the image transform of image pickup lens 10L input by image pick-up element 11 if carries out image is picked up the recording start operation.Then, from image pick-up element 11 with by the form output simulation picking up image signal of the primary signal (raw signal) of the formed Baeyer of three primary colors (bayer) array of red (R), green (G) and blue (B) as with the synchronous signal of timing signal from timing signal generation portion 12.To pretreatment portion 13 the simulation picking up image signal of output is provided, carries out such as the preliminary treatment of defect correction, and resulting analog picture signal is provided to data transaction portion 14 with the γ correction through it.

Data transaction portion 14 is for conversion into digital picking up image signal or the YC data that are made up of luminance signal component Y and color difference signal component Cb/Cr with the simulation picking up image signal of the primary signal form of input.To write image storage unit 4 from the digital picking up image signal of data transaction portion 14 according to operate the image pickup instruction that input unit 3 received through the user.

Particularly, be to pick up instruction if operate image pickup instruction that input unit 3 received by depressing the static images that shutter release button causes through the user, then will write image storage unit 4 from the digital picking up image signal of data transaction portion 14.In this case, aforesaid a plurality of frames from the digital picking up image signal of data transaction portion 14 that will superpose are each other write first frame memory to the N frame memory 41 to 4N (the static images number of N) of image storage unit 4 for superposeing.

In the present embodiment, the view data of first frame when depressing shutter release button writes first frame memory 41 as the view data of target frame.Then, with second two field picture and subsequently two field picture data as a reference the view data of frame write second frame memory to the N frame memory 42 continuously to 4N.

After a plurality of images with different frame write image storage unit 4, read the view data of target frame and the view data of reference frame by global motion vector calculating part 15.Then, global motion vector calculating part 15 is carried out detection, the Calculation of Reliability of local motion vector LMV, the calculating of global motion and the calculating of global motion vector GMV of the local motion vector LMV that is described below.

In the present embodiment, the information of the view data REFv of the view data TGv of global motion vector calculating part 15 output global motion vector GMV, target frame and reference frame.

To motion compensated picture generation portion 16 information from the view data REFv of the global motion vector GMV of global motion vector calculating part 15 and reference frame is provided.Motion compensated picture generation portion 16 will be applied to the view data REFv of reference frame with the corresponding processing of global motion (promptly comprise parallelly move, rotation and expansion or the conversion process of shrinking) based on global motion vector GMV, to generate motion compensated image.

Then, the view data TGv from the target frame of global motion vector calculating part 15 is provided, and the view data MCv from the motion compensated image of motion compensated picture generation portion 16 is provided to addition portion 17 to addition portion 17.Addition portion 17 will be in the pixel addition of the corresponding position of view data TGv and the MCv overlap-add procedure with carries out image, and exports the view data MIXv of resulting sum graph picture (for the noise reduction image).

In first frame memory 41 of image storage unit 4 with before the picture number of target frame be rewritten as the view data as target frame from the view data MIXv of the sum graph picture of addition portion 17.

Particularly, follow closely after depressing shutter release button, the view data of the target frame of first frame memory 41 at first is the view data of first frame.Then, if the view data MCv of the motion compensated image of second reference frame and target frame addition, then the view data of the target frame of first frame memory 41 is rewritten the view data MIXv of the result's who becomes addition sum graph picture.

Then, the view data MIXv of sum graph picture is used as the view data of target frame of the view data of the 3rd reference frame.Then, calculate global motion vector GMV as stated similarly through global motion vector calculating part 15, and through addition portion 17 carries out image overlap-add procedure.

Then, re-write first frame memory 41 of image storage unit 4 on the view data of view data MIXv with the result's of addition sum graph picture as the former target frame of the view data of target frame.After this, also carry out similar processing operation for conduct with reference to the 4th frame of image and each frame subsequently.

Therefore, after opening image, in first frame memory 41 of image storage unit 4, write the noise reduction image of all N that will superpose frames that superposeed in the overlap-add procedure of carries out image as N with reference to image.

Then, through system bus 2 to static images coding and decoding portion 18 provide in first frame memory 41 that is stored in image storage unit 4 as the view data MIXv of the sum graph picture of the noise reduction image of stack result and through the 18 conversion encoding and decoding of static images coding and decoding portion.To be recorded in from the dateout of static images coding and decoding portion 18 on the recording medium such as DVD (digital versatile disc) or hard disk of record and regenerating unit unit 5.In the present embodiment, static images coding and decoding portion 18 handles static images carries out image compressed encoding according to JPEG (JPEG) system.

In addition, in the static images image pickup mode, before pressing shutter release button, first frame memory 41 through image storage unit 4 provides the view data from data transaction portion 14 to conversion of resolution portion 19.Then, view data is converted into the data of predetermined resolution through conversion of resolution portion 19, then, the data that provide through conversion to NTSC (national television system committee) encoder 20.NTSC encoder 20 is converted into view data the reference colour picture signal of NTSC system.Then, to can the reference colour that obtain picture signal being provided by the monitor display unit 6 that forms such as LCD (liquid crystal display) panel.Then, at the monitoring picture that shows on the display screen of monitor display unit 6 under the static images image pickup mode.

Begin operation in response to the regeneration of operating input unit 3 through the user; The view data of the static images of playback record on the recording medium of record and regenerating unit unit 5; And the view data of this static images is provided, and the view data of static images is decoded with regeneration through this static images coding and decoding portion to static images coding and decoding portion 18.Then; Unshowned buffer storage through image storage unit 4 provides the view data of the static images that is reproduced decoding to NTSC encoder 20, and the view data of this static images is for conversion into the reference colour picture signal of NTSC system through NTSC encoder 20.Then, the reference colour picture signal is provided, and on the display screen of watch-dog display unit 6, shows its reproduced picture to monitor display unit 6.

Although it should be noted that among Fig. 1 not illustrate, can derive output image signal to the outside through output end of image from NTSC encoder 20.

It should be noted that in the present embodiment,, can omit static images coding and decoding portion 18 though pass through static images coding and decoding portion 18 with the compressed format recording image data, thereby not with the compressed format recording image data.

In addition, can constitute above-mentioned global motion variable calculating part 15 and motion compensated picture generation portion 16 by hardware.In addition, can use DSP (digital signal processor) to constitute global motion variable calculating part 15 and motion compensated picture generation portion 16.In addition, can replace global motion variable calculating part 15 and motion compensated picture generation portion 16 through software processes by CPU1.

Similarly, can also constitute addition portion 17 with hardware or with DSP.In addition, can also replace addition portion 17 through software processes by CPU 1.This also is applied to static images coding and decoding portion 18 similarly.

[global motion vector calculating part 15]

In the present embodiment, global motion vector calculating part 15 uses above-mentioned sad value execution block matching treatment with reference to Fig. 2 A～Fig. 7, to carry out local motion vector detection.Yet, it should be noted that in the present embodiment, constitute global motion vector calculating part 15 by the hardware that is described below, and calculate local motion vector through hierarchical block matching treatment and interpolation processing.

In addition, as stated, global motion vector calculating part 15 also calculates the reliability index in each local motion vector.

In addition, global motion vector calculating part 15 only uses those local motion vectors with high reliability to calculate global motion.Then, global motion vector calculating part 15 is the global motion vector of unit according to the global motion calculating that calculates with the piece.

< hierarchical block matching treatment >

In the ordinary movement vector detection process in the conventional bar coupling of the prior art; In the hunting zone be that unit (be unit with a pixel promptly or be unit with a plurality of pixels) moves reference block, and calculate sad value at the reference block at each place, shift position with the pixel.Then, from the sad value that calculates by this way, detect sad value, and detect motion vector based on the reference block locations that shows minimum sad value into minimum.

In addition, in the present embodiment, because a frame is divided into a plurality of, so through continuous switch target piece and reference block and to above-mentioned matching treatment of whole image execution.Thereby carry out the calculating of the local motion vector LMV of all object block in target frame.

Yet; Handling the problem that exists in aforesaid this motion vector detection of the prior art is; Owing in the hunting zone, be that unit moves reference block with the pixel; Increase so be used to calculate the number of times of the matching treatment of sad value, thereby increased the matching treatment number of times pro rata with the hunting zone.In addition, motion vector detection of the prior art is handled and is also had another problem, and promptly the capacity of SAD table also increases.

If it is very big that the high pixelation of consideration still image and the size of the high Qinghua of live image (more high definition) and piece image become, this problem is just especially serious.Motion vector detection of the prior art is handled and is also had the problem that number of times increases and bus bandwidth must increase that gets into video memory through system bus 2.

Consider the problems referred to above, in the present embodiment, carry out the hierarchical block coupling; Wherein, Target image or target frame and reference picture or reference frame at first reduce size, with image and the intermediate image of preparing to dwindle, then; In the search of next stage under the situation of piece matching result of reflection previous stage, carry out motion-vector search through the piece coupling with the order of the image, intermediate image and the original image that dwindle.

Through carrying out the hierarchical block coupling, to carry out the calculating of local motion vector effectively than calculating in a small amount with than the short processing time.It should be noted that the image that dwindles hereinafter is called reduced plan, intermediate image hereinafter is called mid-plane, and is not that the original image of scaled version hereinafter is called cardinal plane.

Figure 11 shows the hierarchical block coupling.With reference to Figure 11; Shown in instance in; Cardinal plane target frame 201 and cardinal plane reference frame 301 are decreased to 1/a1/b (1/a and 1/b are minification, wherein, a＞1 and b＞1) dimensionally to generate reduced plan target frame 211 and reduced plan reference frame 311 respectively.

Then, cardinal plane target frame 201 and cardinal plane reference frame 301 are decreased to 1/b to generate mid-plane target frame 221 and mid-plane reference frame 321 respectively.

Although arbitrary proportion all can be applied to reduced plan and mid-plane with respect to cardinal plane, it can suitably be set to 1/2～1/8 times, and promptly the conversion imaging prime number is 1/4～1/64 times.Noting, in the instance of Figure 11, is 1/4 with respect to the minification of the reduced plan of mid-plane, that is, and and a=4, and be 1/4 with respect to the minification of the mid-plane of cardinal plane, that is, and b=4.

In addition, for the generation of reduced plan and mid-plane, can use any means.Yet; If use only in response to the method for the minification that generates reduced plan or mid-plane to the pixel sampling of original image; Then generate reflecting component (reflection component), detected motion vector departs from correct motion vector probably in ground floor (reduced plan).Therefore, the low pass filter that will have the cut-off frequency bandwidth that is suitable for certain minification usually at first is applied to original image, then, carries out the sampling that is suitable for this minification.

In the present embodiment, in the pixel that comprises owing to those pixels that disappear with the certain proportion sampling, calculate the brightness value that average brightness and this average brightness are used as reduced plan pixel or mid-plane pixel.Particularly, under the situation that 1/a dwindles, calculate the brightness value that average brightness and this average brightness in the square area of a * a pixel are used as reduced plan pixel or mid-plane pixel.In this case,, generate reduced plan by mid-plane then, also can obtain and directly generate the identical result that is obtained under the situation of reduced plan by raw frames even at first form mid-plane.Therefore, this method is higher aspect efficient.

It should be noted that in the time downscaled images will being generated minification in the horizontal direction can be identical with minification in vertical direction or also can differ from one another as above-mentioned situation.

After generating reduced plan and mid-plane with above-mentioned this mode, reduced plan object block 212 is set to reduced plan target frame 211 and reduced plan hunting zone 313 is set to reduced plan reference frame 311.

Then, through 401 pairs of a plurality of reduced plan reference blocks 312 execution block matching treatment in reduced plan hunting zone 313 of reduced plan device for detecting motion vector, go out the reduced plan reference block locations of minimum sad value with detected representation.Then, the detection based on regeneration plane reference block detects reduced plan motion vector MVs.

In this example; Reduced plan device for detecting motion vector 401 is carried out the processing that is used for piece matching treatment unit; This piece matching treatment unit is the piece of the size of reduced plan object block 212, that is, and and the piece of the line number of pixel count in the horizontal direction * in vertical direction.

After the calculating of reduced plan motion vector MVs finished, mid-plane object block 222 was set up on the mid-plane target frame 221, and mid-plane target frame 221 sizes equal reduced plan target frame 211 and multiply by a.

In the instance of Figure 11, object block execution block matching treatment in the middle of 402 pairs of the mid-plane device for detecting motion vector, this intermediate objective piece is the piece with the piece matching treatment unit same size of reduced plan vector detection device 401.The piece of same size is the line of equal number of the piece of identical pixels number and the pixel that comprises equal number in the horizontal direction * in vertical direction.

Under the situation of this instance because reduced plan has the size of the 1/a of mid-plane, with the zone of reduced plan object block 212 corresponding mid-plane target frame in the quantity of included mid-plane object block 222 be a.Therefore, all mid-plane object block 222 is set to the piece matching treatment object of mid-plane device for detecting motion vector 402.

Then, in mid-plane reference frame 321, the mid-plane hunting zone 323 that concentrates on reduced plan motion vector MVc place is set with the size that doubly equates with a of reduced plan reference frame 311.Then; To carrying out above-mentioned matching treatment by a plurality of mid-plane reference blocks 322 of device for detecting motion vector 402 in mid-plane hunting zone 323; Go out the mid-plane reference block locations of minimum sad value with detected representation, to detect mid-plane motion vector MVm.

In the hunting zone of each mid-plane object block of mid-plane device for detecting motion vector 402 in being arranged on mid-plane hunting zone 323 to each the execution block matching treatment in the object block of middle plane, thereby detect the motion vector of each mid-plane object block.Then, detected representation goes out that the motion vector MVm as mid-plane in a plurality of motion vectors of minimum sad value, that is, and and as the mid-plane motion vector.

After the calculating of reduced plan motion vector MVs finishes, cardinal plane object block 202 is arranged on the b cardinal plane target frame 201 doubly that size equals mid-plane object block 221.

In the instance of Figure 11; Cardinal plane device for detecting motion vector 403 is also carried out the piece matching treatment that is used for the processing unit piece; Piece in this processing unit piece and device for detecting motion vector 401 and 402 has equivalent size; That is the identical line number on the same pixel number * vertical direction on same pixel number=horizontal direction.

Then, be that unit obtains mid-plane motion vector MVm with aforesaid processing unit piece.Therefore, as the quantity of the cardinal plane object block 202 of the cardinal plane target frame 201 of the object of cardinal plane device for detecting motion vector 403 be set to and piece (be processing unit piece) that target reduced plan fast measure-alike pointed like Figure 11 bend quantity b doubly.

On the other hand, equal in the b cardinal plane reference frame 301 doubly of mid-plane reference frame 321 in size, the resultant vector that is provided with reduced plan motion vector MVs and mid-plane motion vector MVm is the cardinal plane hunting zone 303 at center.Carry out above-mentioned matching treatment by a plurality of cardinal plane reference blocks in 403 pairs of cardinal plane hunting zones 303 of cardinal plane device for detecting motion vector 302, go out the position of the cardinal plane reference block of minimum sad value with detected representation, to detect cardinal plane motion vector MVb.

Processing unit piece with same size is that unit obtains reduced plan motion vector MVs and mid-plane motion vector MVm.Therefore, be provided with resultant vector with reduced plan motion vector MVs and mid-plane motion vector MVm be the center cardinal plane hunting zone 303 than the zone that comprises b cardinal plane object block 202 more greatly.

In the hunting zone of the cardinal plane object block of cardinal plane device for detecting motion vector 403 in being arranged on cardinal plane hunting zone 303 to b cardinal plane object block 202 execution block matching treatment, thereby carry out the detection of the motion vector of cardinal plane object block.Then, detected representation goes out in a plurality of motion vectors of minimum sad value one as cardinal plane motion vector MVb, that is, and and the cardinal plane motion vector of cardinal plane.

Then; As the resultant vector of the reduced plan motion vector MVs that confirms with above-mentioned this mode, mid-plane motion vector MVm and cardinal plane fortune vector MVb, detect the local motion vector LMV of the cardinal plane object block between cardinal plane target frame 201 and the cardinal plane reference frame 301.

In continuous switch target piece and reference block, the All Ranges of target frame and reference frame is carried out above-mentioned this hierarchical block matching treatment, thereby be all a plurality of local motion vector LMV of unit calculating with a plurality of object block of setting in the target frame.

In the instance of Figure 11, device for detecting motion vector 401,402 and 403 is actually 1 device, and the reference block of just reading with the object block of from image storage unit 4, reading and importing and in the hunting zone is different.

It should be noted that in cardinal plane target frame 201 and be configured to obtain to carry out the conversion of object block in the following manner under the situation of local motion vector LMV of all cardinal plane object block 202.Particularly, in the horizontal direction, when moving reduced plan continuously, the reduced plan object block is set on reduced plan with quantity corresponding to the pixel count on the horizontal direction according to ratio 1/a and 1/b.Meanwhile, in vertical direction, when moving reduced plan continuously, the reduced plan object block is set with quantity corresponding to the line number on the vertical direction according to ratio 1/a and 1/b.

Yet,, can take following countermeasure from the purpose of confirming global motion vector by a plurality of local motion vector LMV.Particularly, can with the reduced plan object block be set in the horizontal direction with vertical direction on move continuously, thereby obtain with at the relevant local motion vector LMV of the cardinal plane object block of skipping the position of cardinal plane target frame 201.

It should be noted that also can omit mid-plane only carries out above-mentioned hierarchical block coupling through reduced plan and two layerings of cardinal plane, perhaps can comprise a plurality of intermediate layers that are used for different mid-planes.But,, then take every caution against error if very high the making of minification comprised that in same cell block moving image picks up object and background.Particularly, the motion vector that should be detected as the different motion vector at first is used as single motion vector and handles, and owing in succeeding layer, can not recover, carries out the selection of minification so take every caution against error.

[calculating the reliability of local motion vector LMV]

At the image with more a large amount of noises is under the situation of target image, because the noise effect sad value, so usually can not obtain correcting vector.Figure 12 is the photo that comprises the night scene of more a large amount of noises.If camera then obtains this result shown in figure 13 at the motion vector of shaking on the left direction between the image that is picked up when the image of Figure 12 being shown and relating to rotation very in a small amount.Obtain Figure 13 through drawing the reduced plan motion vector that former figure is decreased to 1/8 downscaled images that obtains.

Can be as can be seen from Figure 13, the texture of acquisition is not that the special motion vector of the night sky clearly is various diverse motion.In the hierarchical block coupling, owing to application of low-pass filters when the generation of downscaled images, so noiseproof feature is than higher.Yet shown in figure 13, noise can influence the image that dwindles.

Owing to obtain the local motion vector LMV of cardinal plane through search around the reduced plan motion vector; If dwindle the picture motion vector so move from correct motion vector; It is invalid then recovering; Noise directly influences the reduced plan motion vector, and further upsets this reduced plan motion vector.

Even object images is the captured image that does not have noise fully; If the texture of image is unintelligible; Then when continuously shot images since the minor variations of exterior light or the difference of time for exposure cause gray scale (gradation) to change will be bigger, and detected motion vector usually departs from right value.In addition, though a large amount of trees and artificial building (such as the house) have the textured pattern of a lot of repetitions, even if as the repeat patterns of described this texture just, detected motion vector also possibly depart from right value.

Suppose as described this situation just, attempted only using motion vector computation global motion in the prior art with high reliability.For example, the motion vector of piece that has proposed to carry out the rim detection of target image and confirmed to have sharp edge is as the motion vector with high reliability.In addition, IDCT (discrete inverse cosine transformation) result's of target image DC component and AC component computed reliability have been proposed to use.

Also recommended a kind of method, wherein, used angle detector (corner detector) to detect the characteristic point on the target image, made the motion vector of gained have high reliability as a kind of filter.Also recommended a kind of technology, supposed on reference picture, also to keep the position relation of a plurality of characteristic points, from the combination of a plurality of motion vectors of difference, extracting the motion vector of high reliability.

Yet above-mentioned these technology of the prior art are not supposed the image of strong noise, and obviously invalid for the very high image of noise level.

In the present embodiment, consider above-mentioned situation, adopted a kind of countermeasure to obtain the reliability index value, through its effective reliability of estimated motion vector, even the image under the high-noise environment.

In the present embodiment, first maximum in the correlation between object block and the reference block and the second peaked difference or ratio are used as the exponential quantity of motion vector reliability.In the present embodiment, owing to detect correlation between object block and the reference block as sad value, so first maximum of correlation and second maximum are respectively first minimum value and second minimum value of sad value.

Figure 14 schematically shows the sad value about the SAD table of an object block.In Figure 14, the hunting zone is represented as the two dimensional range on the horizontal direction of image or x direction and vertical direction or y direction, and (that is, with the x direction vertical with the y direction on) got sad value on short transverse.Therefore, the SAD indumentum is expressed as cubic surface.

In piece matching treatment commonly used,, only the minimum of sad value in the SAD table is confirmed as detected object in order to detect motion vector.Yet this minimum of sad value is first minimum value of the sad value in the SAD table, and in Figure 14, and this value hypothesis is by putting 501 represented positions.In Figure 14, detect motion vector MV as (promptly (x=0, y=0)) is to the vector by the minimum value position of putting 501 represented sad values from the initial point of motion.

If consider the perfect condition that does not have noise; Then when a plurality of reference blocks in definite hunting zone and the correlation between the object block; The SAD that is represented by cubic surface shows following state: cubic surface is evenly protruded downwards, and only has the minimum value of a sad value.Yet; Under the image pickup states of reality; Because have not only that light quantity changes, the influence of the motion of movable body etc.; But also have various The noise, so the SAD table of being represented by cubic surface shows evenly the shape of protrusion downwards hardly, but have the minimum value of a plurality of sad values usually.

Therefore, in the present embodiment, detect motion vector MV based on the position of the reference block of first minimum value that shows the minimum that equals sad value.Yet, the minimum value in the sad value except that first minimum value of sad value (being second minimum value of the sad value) index that is used to generate reliability to be detected.In Figure 14, represent first minimum value by the position of point 501 expressions, and represent second minimum value by the position of another point 502 expressions.

If the influence of noise etc. is limited, then the difference between second minimum value of first minimum value of sad value and sad value is very big, and the reliability of the motion vector MV that is detected from first minimum value of the sad value minimum of sad value (promptly from) is very high.On the other hand; Under the another kind of environment that comprises much noise etc.; Difference between first minimum value of sad value and second minimum value of sad value is very little, and can not distinguish in first and second minimum values of sad value which correctly corresponding to motion vector MV.Therefore, reliability is very low.

As stated, in the present embodiment, the difference between second minimum value of first minimum value of sad value (minimum of sad value) and sad value is confirmed as the index of the reliability of the motion vector that is detected.Figure 15 shows the SAD table of the hunting zone among expression Figure 14 on the one dimension axle.In the present embodiment, the difference between second minimum value among Figure 15 and first minimum value (being the minimum of sad value) is confirmed as the exponential quantity Ft of motion vector MV.

It should be noted that in first minimum value that only obtains sad value not obtain under the situation of second minimum value, in the present embodiment, the peak of the sad value in theoretic peak of sad value or the sad value table is confirmed as the reliability index value of motion vector MV.Therefore, confirm that this motion vector of just having described is very high.Yet, because seldom there be the motion vector that does not obtain the piece of second minimum value so can from the evaluation of reliability, get rid of first minimum value that only therefrom obtains sad value in the piece of the type of describing.

Should note; Difference between first minimum value (minimum of sad value) of replacement sad value and second minimum value of sad value, the ratio between second minimum value of first minimum value of sad value (minimum of sad value) and sad value can be used as the exponential quantity Ft of motion vector MV.Yet, in the following description, the difference between first minimum value (minimum of sad value) of use sad value and second minimum value of sad value.

According to the reliability index of the motion vector in the present embodiment and since not have to use as of the prior art such as image border or picture characteristics picture content and only used the correlation between target frame and the reference frame, so very high for the robustness of noise.In other words, obtain to have the reliability index of high-precision motion vector, do not influence and can not receive picture noise.

In addition; In the present embodiment, using difference or ratio between second maximum (second minimum value of sad value) of first maximum (first minimum value of sad value) and correlation of correlation also is to make the reliability index of motion vector in the present embodiment have the reason of very high anti-noise intensity.

Particularly, if noise level raises, even then motion vector is correct, the sad value of motion vector also can raise usually.Therefore, in order to extract motion vector with high reliability to the reliability index value Ft setting threshold of motion vector to carry out under the situation with the comparison process of threshold value, also need change the threshold value of itself in response to noise level.

On the contrary, under the situation of the exponential quantity Ft of the motion vector in using present embodiment, second maximum (second minimum value of sad value) of first maximum of correlation (first minimum value of sad value) and correlation all raises in response to noise level.Therefore, in the difference between second maximum of first maximum (first minimum value of sad value) of correlation and correlation, The noise is cancelled.

In other words, can realize not relying on the threshold process of the fixed value of noise level.This also is applied to the situation that ratio between second maximum (second minimum value of sad value) of first maximum (first minimum value of sad value) and correlation of correlation is used as the exponential quantity of motion vector similarly.

Incidentally, under the very low situation of the contrast of the image of the object piece of wanting execution block coupling, the difference between second minimum value of sad value and the minimum of sad value has the trend that reduces.Therefore, when same number of frames comprises the zone with high-contrast and have another zone of low contrast, if identical threshold value is used to the evaluation of estimate Ix of pricing vector reliability, the zone that then preferential probably selection has high-contrast.

Although this is correct result from the angle of motion vector reliability; But in order to relieve the zone of low contrast to a certain degree; In the present embodiment, the item that is used to alleviate the contrast influence is added into the mathematical operation expression formula of the exponential quantity of the reliability that is used for confirming motion vector.Particularly, confirm the highest brightness value of target frame image and the difference between the minimum brightness value, and the difference of reflection brightness on the exponential quantity of the reliability of motion vector.It should be noted that harmful effect for fear of noise, with application of low pass filters after the view data of target frame, carry out the extraction of maximum brightness and minimum brightness.

In view of mentioned above, provide the calculation expression of exponential quantity Ft in the present embodiment:

Ft＝(Btm2SAD-MinSAD)-(MaxTAR-MinTAR)×Co

... (expression formula 14)

Wherein

Ft: the reliability index value of motion vector

Second minimum value of Btm2SAD:SAD value

The minimum of MinSAD:SAD value (first minimum value)

MaxTAR: the highest brightness value of object block

MinTAR: the minimum brightness value of object block

Co: weight coefficient (≤1)

Should note; Ratio between second maximum of first maximum of correlation and correlation is used as under the situation of motion vector reliability index value; Be quite analogous to preceding text given (expression formula 1), the item that being used to alleviate contrast influences also may be added to reliability index value calculation expression.Yet, in the calculating of the exponential quantity Ft of motion vector, be not in fact to add the item that is used to alleviate the contrast influence, and can omit this.

In the superincumbent description,, certainly, also can confirm motion vector reliability index value similarly about reduced plan motion vector MVs or mid-plane motion vector MVm though only confirm the motion vector reliability index value of cardinal plane motion vector MVb.

[calculating of motion vector and global motion vector GMV]

In the prior art, do not use the reliability index value of above-mentioned motion vector.Therefore, a plurality of local motion vector LMV that confirm about target frame all use identical power to calculate global motion.

On the contrary, in the present embodiment, can obtain the reliability index value Ft separately of a plurality of local motion vector LMV of target frame with mode mentioned above.

Then, can standardize, among the local motion vector LMV each is provided with for example more than or equal to 0 weight coefficient smaller or equal to 1 to the reliability index value of determined a plurality of local motion vector LMV by this way.Then, can not utilize identical power but through using local motion vector LMV, to calculate global motion according to the determined a plurality of power of each weight coefficient.Particularly; When using determined whole a plurality of local motion vector LMV; When calculating with the convergence that begins to be used to calculate global motion, local motion vector LMV utilize be worth corresponding weight coefficient weighting with each reliability index after, use this local motion vector LMV.

Yet, handle for the mathematical operation of simplifying global motion and to reduce mathematical operation load, in the present embodiment, the weight coefficient W two of local motion vector LMV is advanced value become 0 and 1.

Therefore, in the present embodiment, set the threshold value th of the exponential quantity Ft that is used for motion vector, and use the exponential quantity Ft of each motion vector to calculate the weight coefficient W of each local motion vector LMV according to following mathematical operation expression formula:

When Ft＞th, W=1, and

When Ft≤th, W=0 ... (expression formula 15)

Particularly; In the present embodiment; With motion vector reliability index value Ft be used for judging a plurality of local motion vector LMV each reliability and from a plurality of local motion vector LMV, only extract the local motion vector LMV that those have high reliability; Then, only the local motion vector LMV with high reliability that is extracted is used to calculate global motion.

In the present embodiment, because the object block number in the target frame is more relatively, so, also can calculate and have high-precision global motion even using as this instance only extraction to have under the situation of method of local motion vector LMV of high reliability.

It should be noted that hereinafter, described the concrete processing instance that calculates global motion according to a plurality of local motion vector LMV.

Described hereinbefore, from reference to obtaining this local motion vector shown in figure 13 the described image with much noise of Figure 12.Yet; If local motion vector represented on the image of reliability index value to Figure 13 of use according to the motion vector of present embodiment is carried out reliability decision; Thereby only extract those reliability index values and draw piece and motion vector subsequently, then can obtain image shown in figure 16 with the reliability that is higher than threshold value.Consider this point, for piece shown in Figure 16, the local motion vector that can obtain to be in the main true, and do not influenced by noise.

[instance of the hardware configuration of global motion vector calculating part 15]

Global motion vector calculating part 15 is carried out the reliability index value to each object block detection local motion vector LMV, the calculating local motion vector LMV that detects mentioned above, is calculated processing such as global motion and global motion vector GMV.

Figure 17 shows the instance of the hardware configuration of global motion vector calculating part 15.With reference to Figure 17, global motion vector calculating part 15 comprises: object block buffer part 151 is configured to store the pixel data of object block 102; And reference block buffer part 152, be configured to the pixel data of stored reference piece 108;

Global motion vector test section 15 further comprises matching treatment portion 153, is configured to calculate the sad value of the respective pixel of object block 102 and reference block 108.Global motion vector test section 15 further comprises local motion vector calculating part 154, is configured to according to calculating local motion vector from the sad value information of matching treatment portion 153 outputs.Global motion vector test section 15 further comprises: control part 155, motion vector reliability index value calculating part 156, global motion vector mathematical operation portion 157 and contrast calculating part 158.

Contrast calculating part 158 comprises low pass filter 1581, highest brightness value test section 1582 and minimum brightness value test section 1583.

In addition; Although do not illustrate; But in this example, be stored and be retained in the image storage unit 4 by the view data of the reduced plan of the target frame that view data generated of the target frame of original image and reference frame and reference frame and the view data of mid-plane.

Control part 155 is controlled the processing sequence (sequence) of global motion vector calculating parts 15, and control signal is offered the assembly of global motion vector calculating part 15 shown in figure 17.

Object block buffer part 151 is read in the view data of intended target piece from the view data of the target frame of reduced plan, mid-plane or the cardinal plane of image storage unit 4, and under the control of control part 155, view data is offered matching treatment portion 153.

Reference block buffer part 152 is read in the view data of specifying in the matching treatment scope from the view data of the reference frame of reduced plan, mid-plane or the cardinal plane of image storage unit 4 under the control of control part 155.Then, reference block buffer part 152 sequentially will offer matching treatment portion 153 from the view data of the reference block in the view data in the matching treatment scope.

Matching treatment portion 153 receives from the view data of the object block of object block buffer part 151 with from the view data of the reference block of reference block buffer part 152.Then, matching treatment portion 153 under the control of control part 155 to reduced plan, mid-plane and cardinal plane execution block matching treatment.Then, matching treatment portion 153 offers local motion vector calculating part 154 with reference vector (being the positional information of reference block) and piece matching treatment result's sad value.

Local motion vector calculating part 154 comprises first minimum value storage part 1541 of sad value and the second minimum value storage part 1542 of sad value, and carries out the processing of second minimum value of first minimum value that from the sad value from matching treatment portion 153, detects sad value and sad value.

Then, the positional information (being reference vector) of second minimum value of second minimum value of the sad value in the second minimum value storage part 1542 of the positional information (being reference vector) of first minimum value of first minimum value of the sad value in the first minimum value storage part 1541 of local motion vector calculating part 154 sequential update sad values and sad value and sad value and sad value.Local motion vector calculating part 154 is carried out this update processing, and the piece matching treatment of all reference blocks finishes in the matching treatment scope.

Then, when the piece matching treatment finishes, will be in the positional information or the first minimum value storage part 1541 that reference vector is stored in sad value of first minimum value of first minimum value and sad value of sad value of object block of this time point.In addition, positional information or the reference vector with second minimum value of second minimum value of sad value and sad value is stored in the second minimum value storage part 1542.

Then; When the piece matching treatment of all reference blocks in the matching treatment scope finishes; The information (being position information) of the reference vector of local motion vector calculating part 154 detection of stored in the first minimum value storage part 1541 of sad value is as the motion vector in reduced plan, mid-plane and cardinal plane each.Hereinafter will be described the processing operation of local motion vector calculating part 154 in detail.

Local motion vector calculating part 154 in the present embodiment offers control part 155 with reduced plan motion vector MVs as local motion vector LMV when the reduced plan matching treatment.

Control part 155 is confirmed the hunting zone of mid-plane according to the information of reduced plan motion vector MVs.Then, control part 155 offers object block buffer part 151, reference block buffer part 152 and matching treatment portion 153 with control signal, so that they carry out the piece coupling in the mid-plane.

Then, when the matching treatment in the middle plane finished, local motion vector calculating part 154 offered control part 155 with the resultant vector information of reduced plan motion vector MVs and mid-plane motion vector MVm as local motion vector LMV.

Control part 155 is confirmed the hunting zone of cardinal plane according to the resultant vector information of reduced plan motion vector MVs and mid-plane motion vector MVm.Then, control part 155 offers object block buffer part 151, reference block buffer part 152 and matching treatment portion 153 with control signal, so that they carry out the piece coupling in the cardinal plane.

When the matching treatment in the cardinal plane finished, local motion vector calculating part 154 offered global motion vector calculating part 157 with the resultant vector information of reduced plan motion vector MVs, mid-plane motion vector MVm and cardinal plane motion vector MVb as local motion vector LMV.The local motion vector LMV that the 157 interim storages of global motion vector calculating part receive.

In addition, when the matching treatment in the cardinal plane finishes, launch (enable) motion vector reliability index value calculating part 156 by control part 155.The second minimum value Btm2SAD of sad value of minimum MinSAD and the second minimum value storage part 1542 of the sad value of the first minimum value storage part 1541 meanwhile, is provided to motion vector reliability index value calculating part 156 from local motion vector calculating part 154.

In addition, at this moment, the view data of object block is offered highest brightness value test section 1582 and minimum brightness value test section 1583 through low pass filter 1581 from object block buffer part 151.Then, will offer motion vector reliability index value calculating part 156 by highest brightness value MaxTAR and the minimum brightness value MinTAR that highest brightness value test section 1582 and minimum brightness value test section 1583 are detected respectively.

Motion vector reliability index value calculating part 156 uses the information that offers it, and (expression formula 1) that provides according to preceding text come the reliability index value Ft of calculating kinematical vector.Then, motion vector reliability index value calculating part 156 offers global motion vector calculating part 157 with the reliability index value Ft that calculates.The reliability index value Ft that global motion vector calculating part 157 comes interim storage to be imported with the relation that is associated with the local motion vector LMV that at this moment offers it.

After above-mentioned a series of processing of all object block finished in target frame, control part 155 offered global motion vector calculating part 157 with control command signal, handled with the mathematical operation of beginning global motion.

In the present embodiment; Global motion vector calculating part 157 is at first according to the control command signal from control part 155; The motion vector reliability index value Ft of use to store with the corresponding relation of local motion vector LMV, (expression formula 2) that provides according to preceding text carried out the judgement of the reliability that is stored in a plurality of local motion vector LMV wherein.Then, global motion vector calculating part 157 only extracts those local motion vector LMV that shows high reliability.

Then, global motion vector calculating part 157 only uses the local motion vector LMV of the extraction with high reliability to carry out the mathematical operation processing of calculating global motion.Then, the global motion that global motion vector calculating part 157 usefulness calculate calculates global motion vector GMV, and resulting global motion vector GMV is offered motion compensated picture generation portion 16.

The 16 view data REFv for the reference frame that sends through global motion vector calculating part 15 of motion compensated picture generation portion utilize global motion vector GMV to carry out and the corresponding conversion process of global motion, to generate motion compensated image.Then, through addition portion 17 motion compensated image that is generated is superimposed upon on the view data of target frame.

Owing to generate global motion and global motion vector GMV in the present embodiment according to the local motion vector LMV with high reliability, so it has very high precision, and the noise reduction image that is obtained through stack has good quality.

[the processing operation of local motion vector calculating part 154]

In order to detect the minimum value of sad value; Local motion vector calculating part 154 in the present embodiment is confirmed position Po as judging object-point, and carries out the sad value of judging Po place, object-point position and by near the judgement object-point position Po that frame of broken lines centered among Figure 18 and the comparison between 8 sad values on every side.Then, local motion vector calculating part 154 judges whether the sad value of this judgement object-point is by the minimum (hereinafter is called local minimum) in 9 sad values in the frame of broken lines institute region surrounded.

Then, if confirm to judge that the sad value of object-point is a local minimum, then the minimum value of local motion vector calculating part 154 sad value that will judge object-point and the sad value of being stored at this moment compares.Then, if judging the sad value of this judgement object-point, local motion vector calculating part 154 is lower than the minimum value of the sad value of being stored this moment, the minimum value of then coming updated stored sad value therein with the minimum sad value in new detected part.

As the structure that is used to detect the minimum sad value in above-mentioned this part, design local motion vector computation portion 154 is used to store the SAD table of sad value with reduction the scale of buffer storage.Particularly; When Where topical motion vector computation portion 154 is unit searching for reference piece with a pixel, preparation can storage of water the buffer storage of sad value+3 sad value of two row of the size of the object block buffer storage of storing the SAD table of sad value as shown in figure 18 being used to square upwards.

Shown in figure 18, if, then can judge and the local minimum at Po place, object-point position with the sad value+3 sad value write buffering memory of two row of object block size on the horizontal direction.

For the minimized in size with buffer storage, shown in figure 18, the sad value of new input is overwritten at the memory location Pa that has stored the old sad value that is not used further to minimum value evaluation or local minimum detection.Particularly,, but do not write memory location Pb, but utilize the memory location Pa that does not re-use again, thereby restrained the increase of memory hardware scale although should the sad value of newly importing be write memory location Pb shown in Figure 180 according to order.

It should be noted that local motion vector calculating part 154 also comprises the first minimum value storage part 1541 mentioned above and the second minimum value storage part 1542 except that being used to store the buffer of local minimum.

Above-mentioned this series of processes is a basic handling, and this basic handling is applied to first minimum value and second minimum value, with the minimum value of detection sad value and second minimum value of sad value.

In the present embodiment; Though 154 pairs of reduced plans of local motion vector calculating part, mid-plane and cardinal plane are carried out identical operations, local motion vector calculating part 154 detects local motion vector LMV and calculates the reliability index value of local motion vector LMV on cardinal plane.Therefore, concerning cardinal plane, only need second minimum value of sad value, and can omit the calculating and the storage of second minimum value of the sad value on reduced plan and the mid-plane.

Figure 19 shows the flow process of carrying out the detection processing operation of first minimum value and second minimum value through local motion vector calculating part 154.

With reference to Figure 19, in step S101, local motion vector calculating part 154 at first obtains sad value from matching treatment portion 153.Then, in step S102, local motion vector calculating part 154 will judge that the sad value at Po place, object-point position and 8 sad values judging Po location about place, object-point position compare mutually.Then, in step S103, local motion vector calculating part 154 confirms to judge based on comparative result whether the sad value at Po place, object-point position is local minimum.

If judge that in step S103 the sad value of the position around the object-point position Po is not a local minimum, then handle and return step S101, to carry out obtaining of next sad value.

On the other hand; If judge that in step S103 the sad value at Po place, object-point position is a local minimum; Then in step S104, first minimum value and second minimum value that local motion vector calculating part 154 will be stored in sad value wherein compare with the sad value of judging Po place, object-point position.

Then, in step S105, local motion vector calculating part 154 confirms to judge whether the sad value at Po place, object-point position is lower than first minimum value of the sad value that is stored in wherein.Then; If confirm to judge that the sad value at Po place, object-point position is lower; Then in step S106; The sad value that local motion vector calculating part 154 usefulness are stored in the first minimum value storage part 1541 upgrades the second minimum value storage part 1542, and will judge that the sad value at Po place, object-point position is stored in the first minimum value storage part 1541.

Then, in step S109, local motion vector calculating part 154 judges whether accomplished the computing to the sad value of all reference blocks through object block.If judge and also do not accomplish computing, then handle and return step S101, obtain next sad value.On the other hand, if in step S109, judge and accomplished computing to the sad value of all reference blocks through object block, local motion vector calculating part 154 end process programs then.

On the other hand; If confirm in step S105 to judge that the sad value at Po place, object-point position is equal to or higher than first minimum value of the sad value that is stored in wherein; Then in step S107, local motion vector calculating part 154 confirms to judge whether the sad value at Po place, object-point position is lower than second minimum value of the sad value that is stored in wherein.If confirm to judge that the sad value at Po place, object-point position is lower, then in step S108, local motion vector calculating part 154 usefulness judge that the sad value at Po place, object-point position updates stored in the sad value in the second minimum value storage part 1542.

Processing advances to step S109 from step S108, and local motion vector calculating part 154 judges whether accomplished the computing to the sad value of all reference blocks.If judge and also do not accomplish computing, then handle and return step S101, obtain next sad value.On the other hand, object block has been accomplished the computing to the sad value of all reference blocks, then handling procedure end if in step S109, judge.

On the other hand; If confirm in step S107 to judge that the sad value at Po place, object-point position is not less than second minimum value of the sad value that is stored in wherein; Then handle and advance to step S109; Wherein, local motion vector calculating part 154 judges whether object block has been accomplished the computing to the sad value of all reference blocks.If judge and also do not accomplish computing, then handle and return step S101, obtain next sad value.On the other hand, object block has been accomplished the computing to the sad value of all reference blocks, then handling procedure end if in step S109, judge.

Process chart shown in Figure 19 can be applied to reduced plan, mid-plane and cardinal plane similarly.In this case; For reduced plan and mid-plane; Detect the sad value finally be stored in the first minimum value storage part 1541 minimum MinSAD, and detect the corresponding reference vector respectively as reduced plan motion vector MVs and mid-plane motion vector MVm as sad value.Then, output minimum MinSAD and reduced plan motion vector MVs and mid-plane motion vector MVm.

On the other hand, for cardinal plane, detect the sad value finally be stored in the first minimum value storage part 1541 minimum MinSAD, and detect the corresponding reference vector as cardinal plane motion vector MVb as sad value.Then, the minimum MinSAD and the cardinal plane motion vector MVb of output sad value.In addition; For cardinal plane, the sad value (being second minimum value of sad value) that finally is stored in the sad value (being minimum MinSAD) in the first minimum value storage part 1541 and finally be stored in the second minimum value storage part 1542 is offered motion vector reliability index value calculating part 156.

[the processing operation of global motion vector calculating part 157]

< affine transformation of expansion >

In the present embodiment, calculate with a large amount of local motion vector LMV or estimate global motion.Then, with the global motion that calculates, calculate global motion vector GMV or estimate global motion.In this case, use the method that shows global motion through affine transformation.Yet in the present embodiment, use is as the affine transformation of the expansion of the modification of the affine transformation commonly used of prior art.

Managing to carry out " oblique distortions " timing; When being about to look like to return to through the represented rectangular image of the dotted line of Figure 20 through represented this " oblique distortions " ladder diagram of the solid line of Figure 20; Half should be along vertical axis below the ladder diagram picture; When promptly enlarging through the represented y axle horizontal of arrow, the ladder diagram picture top half should level shrink.

As noted before, in affine transformation commonly used, be used for the parameter a of affine transformation, b, c, d, e and f, relevant level enlarges and the parameter of contraction is a.Because parameter a is a fixed value,, affine transformation commonly used proofreaies and correct so can not being used for " oblique distortions ".

Therefore, if write parameter renegotiation as q0y+r0, thereby its value can be along changing like the represented y axle of the downside of Figure 20, and then, in response to vertical axis, affine variation becomes expansion or contraction ratio linear change in the horizontal direction.In other words, parameter a replacement is become the function of the y axle variable y that is used for " oblique distortions " correction.Here, for " oblique distortions " correction in Figure 20, parameter can be linear function, that is, and and function q0y+r0.

Although the affine transformation consideration is necessary to consider " oblique distortions " that the rotation owing to yaw axis produces proofreaied and correct and " oblique distortions " that the synthetic rotation owing to pitch axis and yaw axis produces proofreaied and correct to because " oblique distortions " that the rotation of above-mentioned pitch axis produces proofreaied and correct in Figure 20.

Therefore, in the present embodiment,, expand above-mentioned notion with reference to Figure 20.Particularly, allow to enlarge or contraction, through replacing parameter a, b, d and e like expression formula 7 represented pnx+qny+rn (n=0,1,2,3) by Figure 21 in the direction vertical with arbitrary axis in order to make affine transformation.

Here, relevant at affine parameter a, b, d and e with the conversion of image, and affine parameter c and f respectively with image mobile relevant on left and right directions (to the left or to the right) and above-below direction (up or down).Therefore, affine parameter c and f do not replace becomes function.

When expression affine parameter a as expression formula 7, b, d and e, relate to and amount to 14 parameters, and seem that the affine transformation complexity of expanding is higher.Yet if with after expression formula 7 expansion, the arrangement, it has the form of the expression formula 8 of using the Figure 22 that amounts to 12 parameters.

If use this expression formula of the affine transformation of expansion,, can derive separating of a plurality of local motion vectors uniquely through least square method although then expression formula is complicated.Because this technology is the technology identical with affine transformation, so in Figure 23 to Figure 27, only final result is provided as expression formula 9 to expression formula 13 respectively.

Note, in the present embodiment, become function although will participate in all affine parameter a, b, d and the e replacement of image transform, if image transform only takes place on specific direction, then can be only with the alternative function that becomes of the parameter of concerned direction.For example, in order only to consider, only parameter a replacement is become as at the function shown in the downside of Figure 20 in the shown conversion of the upside of Figure 20.

In addition, although in above-mentioned instance, because the hypothesis linear deformation, so the function that is used to replace is a linear function, if the assumed curve distortion, then function can be quadratic equation or higher-order function.

[using the calculating of the global motion of the affine transformation of expanding]

As the technology that from unspecific a large amount of vectors, derives most preferred global motion, the method shown in the flow chart of present embodiment use Figure 28 and Figure 29.According to this method, from piece, get rid of gradually when picking up the motion vector of those pieces that unlikely meet global motion of object such as moving image with high reliability, utilize least square method to make the parameter convergence of global motion.

Processing according to Figure 28 and Figure 29; Adopt the affine transformation of expansion to make from numerous local motion vectors of low vector accuracy, to get rid of the figure that moves to pick up the error vector of object etc., derive the best global motion that comprises " oblique distortions " with feasible mathematical operation cost simultaneously.

Incidentally; Because the affine transformation of expansion comprises a plurality of affine parameters and is flexibly in the present embodiment; So there is such possibility: the convergence mathematical operation of global motion also can be used for moving the erroneous motion vectors of image pickup object or noise, causes getting rid of this erroneous motion vectors.

Therefore, in the present embodiment, the initial stage in convergence mathematical operation loop; Use affine transformation commonly used to come the eliminating of (hereinafter being called error vector) of execution error motion vector; After this, as from Figure 28 and shown in Figure 29, carry out the convergence mathematical operation of the affine transformation of using expansion.Be reduced to through using after affine transformation commonly used can not get rid of the degree of vector of this oblique distortions factor in vector error; This is because in order to use such technology, attempts using the high precision convergence that also can be used for the oblique distortions factor of the affine transformation of expanding.

In addition; In the present embodiment, the maximum that detects vector error is as motion vector (global motion vector) definite by determined global motion in each circulation of convergence mathematical operation and the difference between the detected motion vector (local motion vector LMV mentioned above).

Then; If the maximum of detected vector error is higher than predetermined threshold value; Then continue affine transformation commonly used, if but the maximum of detected vector error is lower than predetermined threshold value, then carry out the convergence mathematical operation of the affine transformation of using expansion.

Now, describe the method for Figure 28 and Figure 29 in detail.

At first, in step S201, motion vector reliability index value Ft and predetermined threshold value that 157 pairs of global motion vector calculating parts are stored in a plurality of local motion vector LMV wherein compare.Then, through result relatively, global motion vector calculating part 157 only selects its motion vector reliability index value Ft to be higher than the object block of predetermined threshold.This processing is corresponding to following situation: as the expression formula that provides with reference to preceding text 15 is described, two

values

1 and 0 are used as weight coefficient W.

Then, in step S202, global motion vector calculating part 157 judges whether be the convergence circuit of carrying out the convergence mathematical operation for the first time.If judge and carry out convergence circuit for the first time, then global motion vector calculating part 157 only uses the local motion vector of selection and uses affine transformation commonly used to derive and perhaps estimate global motion in step S203.In other words, global motion vector calculating part 157 is derived the affine parameter a～f of global motion.

Then, in step S206, global motion vector calculating part 157 calculates the theoretical local motion vector LMVs of the selected block that is used for mathematical operation based on the global motion of deriving.

Then, in step S207,, calculate the error E n between local motion vector LMV that confirms through the piece matching treatment and the theoretical local motion vector LMVs that in step S206, confirms to a plurality of selected blocks each.

For the local motion vector LMV that confirms through the piece matching treatment and the calculating of the error between the theoretical local motion vector LMVs,, then should come correctly to carry out distance calculation through Pythagorean theorem if pay attention to the precision of mathematical operation.Yet, more pay attention to the quick of mathematical operation if compare precision, can be used as approximate distance apart from summation with what confirm between two motion vectors on level and the vertical both direction.

Subsequently, in step S208, global motion vector calculating part 157 uses all error E n that selected block is confirmed to calculate the mean value Eave and the maximum Emax of all errors.Then, in step S209, global motion vector calculating part 157 is judged whether mean value Eave are lower than for whether its predetermined threshold value θ a and maximum Emax are lower than and is its predetermined threshold value θ b.

Do not satisfy condition in step S209 if judge; Then handle and advance to step S211; Global motion vector mathematical operation piece 157 is got rid of among the error E n of the piece of in step S207, confirming from those pieces that will be used to derive global motion, and error E n satisfies the piece of En=Emax.Perhaps, in step S211, global motion vector calculating part 157 detects those pieces that error E n satisfy En >=θ b, and all above-mentioned detected is got rid of from the piece that will be used to derive global motion.

Then, in step S212, whether the rest block number that global motion vector calculating part 157 is judged the result who gets rid of as piece among the step S211 is less than predetermined threshold value θ c.Be not less than threshold value θ c if in step S212, judge the rest block number, then handle and return step S202, be set to the processing of each step that selected block repeats to begin from step S202 with rest block.

If the rest block number is less than threshold value θ c, then owing to not obtaining suitable global motion, so can not the image of object reference frame be used for the stack of the image of present embodiment.Therefore, if in step S212, judge the rest block number less than threshold value θ c, global motion vector calculating part 157 as in step S213, skip subsequently all processing then to reference frame.The processing that global motion vector calculating part 157 finishes its Figure 28 and Figure 29.

On the other hand; If in step S202, judge it is not the convergence circuit of carrying out the convergence mathematical operation for the first time, then global motion vector calculating part 157 judges in step S204 in step S207 whether the maximum Emax of determined error E n is higher than predetermined threshold value θ d.

When the mathematical operation that affine transformation commonly used is used for carry out global motion and with as this mode of above-mentioned step S211 when carrying out the eliminating of error vector, threshold value θ d is set to not get rid of the vector of oblique distortions component.

If the maximum Emax of decision errors En is higher than predetermined threshold value θ d in step S204, then global motion vector calculating part 157 advances to step S203 with processing.In step S203, it uses affine transformation commonly used to derive global motion.In other words, global motion vector calculating part 157 calculates the parameter of the affine transformation of expansion.After this, global motion vector calculating part 157 is carried out the processing of each step that begins from above-mentioned steps S203.

If the maximum Emax of decision errors En is lower than predetermined threshold value θ d in step S204, then global motion vector calculating part 157 only uses the local motion vector LMV of selected block and the affine transformation of using expansion to derive global motion in step S205.After the processing in step S205, carry out the processing of each step that begins from step S206.

Then; If the maximum Emax that the mean value Eave of decision errors En is lower than threshold value θ a and error E n in the step S209 of Figure 28 is lower than threshold value θ b; Then in step S210, global motion vector calculating part 157 judges that mathematical operation restrains, and the global motion of putting between finally confirming at this moment.After this, handling procedure finishes.

It should be noted that in step S211 convergence rate that can be when confirming global motion vector GMV and the balance between the precision are confirmed only to get rid of error E n and are the piece of worst error Emax and still should get rid of those pieces that error E n is higher than threshold value θ b.If precision is preferential, a kind of method before then can adopting so that error block is got rid of one by one, if still convergence rate is preferential, then can adopt a kind of method in back.

It should be noted that in the step S204 of above-mentioned handling procedure, the maximum Emax of error E n is used to use the judgement of the conversion between the mathematical operation of using affine transformation always and the mathematical operation of the using the affine transformation of expanding.Yet, can not only consider the maximum Emax of error E n but also consider error that the conversion of carrying out in step S204 is judged at mean value Eave.

In this case; In step S204; Whether the mean value Eave of decision errors En is higher than predetermined threshold value, if mean value Eave is higher than threshold value, then uses affine transformation actual figure mathematical operations commonly used; If but mean value Eave then uses the affine transformation actual figure mathematical operations of expansion smaller or equal to threshold value.

Perhaps, in step S204, the two judges all whether it is higher than predetermined each threshold value to the maximum Emax of error E n and mean value Eave.Then, when the two all is higher than each threshold value, use affine transformation actual figure mathematical operations commonly used, but be not higher than each threshold value, use the affine transformation actual figure mathematical operations of expansion when the two.

It should be noted that because with the scope of camera shake correction can by the optically-variable focal distance ratio or to set shape (set shape) identification similar, the maximum of oblique distortions component can be by obtaining in the maximum camera amount of jitter, so the identification of threshold value ratio is easier to.

Global motion vector calculating part 157 calculates global motion vector GMV based on the global motion that is calculated with above-mentioned this mode to each object block.Particularly, affine transformation operation parameter a～1 of the expansion of 157 pairs of global motions that calculate of global motion vector calculating part, so that each object block is confirmed motion vector, each object block is corresponding to theoretical local motion vector LMVs.X and y in the expression formula 8 of Figure 22 use the coordinate of the center of each object block.Determined by this way motion vector becomes the global motion vector GMV of each object block.

Then, will offer motion compensated picture generation portion 16 with the global motion vector GMV of the determined object block of above-mentioned this mode.Then, the global motion vector GMV that uses object block of motion compensated picture generation portion 16 is with the motion compensated picture that generates object block and will offer addition portion 17 through the motion compensated picture that generates.

Figure 30 shows in the present embodiment the global motion vector of determined object block from the image of the Figure 12 that comprises much noise.

[second embodiment]

In the above-described embodiments, local motion vector calculating part 154 execution hierarchical blocks mate the local motion vector LMV that calculates on the cardinal plane.Then, 156 calculating of motion vector reliability index value calculating part are with respect to the reliability index value Ft of the local motion vector LMV of cardinal plane.In addition; Global motion vector calculating part 157 uses the reliability index value Ft of the local motion vector LMV of cardinal plane to extract those local motion vectors LMV with high reliability, and uses the local motion vector LMV with high reliability to calculate global motion vector GMV.

Incidentally, through reduced plan motion vector MVs and mid-plane motion vector MVm being multiply by the inverse of the figure image minification factor of cardinal plane, can obtain cardinal plane motion vector MVb.Therefore, in order to calculate global motion vector GMV, maybe uncertain cardinal plane motion vector MVb, but confirm global motion vector GMV through reduced plan motion vector MVs or mid-plane motion vector MVm.

For example, in order to confirm global motion vector GMV through reduced plan motion vector MVs, the local motion vector LMV of the reduced plan that local motion vector calculating part 154 at first calculates, that is, and reduced plan motion vector MVs.

Then, motion vector reliability index value calculating part 156 is with respect to the motion vector MVs computed reliability exponential quantity Ft of the reduced plan that calculates.In addition, global motion vector calculating part 157 uses the reliability index value Ft of the local motion vector LMV of reduced plan to extract those local motion vectors LMV with high reliability.Then, global motion vector calculating part 157 uses the local motion vector LMV with high reliability to calculate global motion and global motion vector GMV.

Use by this way reduced plan or the local motion vector of mid-plane to confirm that global motion vector GMV provides following advantage.

First advantage is: owing to as stated low pass filter is used to generate reduced plan or mid-plane, thus eliminated noise, and the result, the local motion vector of gained is influenced by noise hardly.

Second advantage is: owing to reduced the object block number in reduced plan or the mid-plane; So the quantity of local motion vector reduces, and the reduction of mathematical operation cost, in addition; Owing to handle required time decreased, so can carry out processing with higher speed.

Like the restriction of above-mentioned instance owing to hardware, the matching treatment block unit of reduced plan, mid-plane and cardinal plane has identical size usually.Therefore, compare with the alternative case of only the cardinal plane execution block being mated, the object block number (being the Local Vector number) with reduced plan of little dimension of picture reduces relatively.

Then, confirming through the reduced plan motion vector under the situation of global motion and global motion vector GMV, possibly omit the motion vector detection of middle plane and cardinal plane is handled.And in this, expectation can improve processing speed.

Therefore, particularly, the reduced plan motion vector is being used for confirm that advantage is very significant under the situation of global motion and global motion vector GMV.

Yet, owing to generate reduced plan or mid-plane through the image that dwindles cardinal plane, so should consider that the precision of this reduced plan motion vector or mid-plane motion vector is relatively low.

Therefore, in the present embodiment, under the situation of using reduced plan motion vector or mid-plane motion vector, carry out interpolation processing.Particularly, use the reduced plan reference block locations that the reduced plan motion vector calculate passing through or the mid-plane motion vector that is calculated represent or the sad value of near reduced plan reference block the mid-plane reference block locations or mid-plane reference block and the positional information of sad value to carry out interpolation processing.Through this interpolation processing, carry out the reduced plan motion vector of pixel precision or the detection of mid-plane motion vector.Be example with the Regeneration Treatment under the situation of reduced plan below, described interpolation processing reduced plan.

For example, all under the situation that is contracted to 1/4 reduced plan execution block coupling, the reduced plan motion vector is the motion vector of 4 pixel precisions in the horizontal and vertical directions.Yet significantly, in the cardinal plane reference frame, the cardinal plane motion vector MVb of 1 pixel precision is present in through reduced plan motion vector MVs being increased near the motion vector that n doubly obtained.

Therefore; Under the situation of the minimum sad value 601 on definite reduced plan shown in figure 31; Can consider to use near the minimum sad value 601 a plurality of sad values (for example; Use 4 sad values 602,603,604 and 605 that on the direction of upper and lower, left and right, are close to respectively with minimum sad value 601) carry out interpolation processing, to detect the motion vector of 4 pixel precisions.In this case, required interior slotting multiplying power is 4 times.

For example, can consider to use conic section to carry out tabular interpolation, with through being the motion vector that reduced plan SAD that unit has carried out matching treatment shows the calculating pixel precision for example with n pixel to SAD.In this case, be to use linearity, three times or the approximate interpolation of luminance curve more although can not use the approximate interpolation of conic section, in this example, use conic section to be similar to interpolation according to the equilibrium between precision and the hardware configuration.

In the approximate interpolation of this conic section; Shown in figure 31, use minimum Smin and near a plurality of sad values minimum Smin position (hereinafter is called contiguous reduced plan sad value) through the sad value in the represented reduced plan SAD table of the reduced plan motion vector of n pixel precision.In this example, use 4 contiguous sad value Sx1, Sx2 and Sy1, the Sy2 of contiguous minimum Smin position on directions X or horizontal direction and Y direction or vertical direction.

Shown in figure 32, the contiguous reduced plan sad value Sx1 and the Sx2 at the minimum Smin of reduced plan sad value and two neighbor point places on directions X or horizontal direction is used to use second approximation curve 700.Particularly, confirm to pass the contiguous reduced plan sad value Sx1 at minimum Smin and two neighbor point places on directions X or horizontal direction and the second approximation curve 700 of Sx2.Thus, the coordinate that curve of approximation 700 is got minimum value becomes, and reduced plan motion vector or the X coordinate Vx of high accuracy reduced plan motion vector of minimum SXmin of the sad value of pixel precision is provided.Provide the expression formula that be used for the approximate interpolation of conic section this moment, like following expression 16:

SXmin＝1/2×(Sx2-Sx1)/(Sx2-2Smin+Sx1)

... (expression formula 16)

The reduced plan sad value that the X coordinate that on the SAD table, adopts at the minimum SXmin of the sad value of the pixel precision of confirming according to calculation expression (expression formula 16) becomes pixel precision presents the X coordinate Vx of minimum.

Can realize the division in the calculation expression (expression formula 16) through repeatedly carrying out subtraction.For example, if the pixel precision that will use be the precision of 1/4 pel spacing of reduced plan, then can only just can realize above-mentioned division through twice subtraction.Therefore, circuit scale is less and the mathematical operation time is shorter, and can realize than the second approximation curve interpolation complicated the very approaching performance of the cubic curve interpolation of Duoing.

Similarly, use the minimum Smin of reduced plan sad value and on Y direction or vertical direction with the contiguous reduced plan sad value Sy1 of contiguous two points of minimum Smin and Sy2 with application second approximation curve.Therefore, the second approximation curve Y coordinate that the presents minimum value SYmin place sad value that becomes pixel precision shows the Y coordinate Vy of minimum.Provide the expression formula of the approximate interpolation of conic section of this moment through following expression (expression formula 17):

SYmin＝1/2×(Sy2-Sy1)/(Sy2-2Smin+Sy1)

... (expression formula 17)

Approximate through by this way directions X and Y direction being carried out twice conic section, confirm high accuracy (being pixel precision) the reduced plan motion vector (Vx, Vy).

In the description formerly; Though the reduced plan sad value of two points that used the minimum of reduced plan sad value and on directions X or horizontal direction and Y direction or vertical direction, be close to minimum can be for more than two in the quantity of the contiguous reduced plan sad value of different directions.In addition, replace the application of the conic section on directions X and Y direction, for example, conic section can be applied to incline direction.In addition, except that X and Y direction, curve of approximation can also be employed in an inclined direction.

Figure 33 shows through using above-mentioned this device and handling procedure, and the value that can show through the SAD of n pixel unit precision obtains the vector detection result of pixel precision.The axis of abscissas of Figure 33 is represented the interpolation multiplying power, and is illustrated in and should sets several times resolution on the one dimension direction.Because SAD table is two dimension, thus the area of table with square ratio reduce.Yet, owing to the error that is caused by interpolation only increases with the degree of linearity, so can approve the serviceability of interpolation technique.

[other embodiment and variation]

In the above-described embodiments; Though the situation of the motion vector reliability that is detected when applying the present invention to judge the image pickup of still picture; But also possibly apply the present invention to another kind of situation naturally, the reliability of the motion vector that is detected when promptly judging the image pickup of motion picture.

In addition, in the above-described embodiments, though sad value is detected as correlation, correlation is not limited to sad value naturally.

In addition, in the above-described embodiments, a plurality of captured images that in image storage unit 4, obtain are carried out the motion detection processing of still picture and the overlap-add procedure of image.Yet a plurality of images of image pickup object are the same in the time of also can be with the image pickup of motion picture to be handled in real time.

The image information that it should be noted that the detected object of motion vector is not limited to captured image information.

In addition, in the above-described embodiments, carry out the judgement of the reliability of motion vector based on motion vector reliability index value.Yet; Not only difference between second maximum of first maximum of correlation and correlation or ratio can be used to carry out the judgement of reliability, and present the first peaked reference vector of correlation and present the judgement that alternate position spike between the second peaked reference vector of correlation also can be used to carry out reliability.Perhaps, the 3rd maximum of correlation or more high-order maximum or present other position distribution of this peaked reference vector can be by further with reference to carrying out the judgement of reliability.

It should be noted that in the above-described embodiments in order to reduce the mathematical operation load when the mathematical operation of carrying out global motion is handled, the binary value with 0 and 1 is used for being worth corresponding weight coefficient W with the reliability index of local motion vector LMV.Yet the weight coefficient W in that the normalization of the reliability index value through local motion vector LMV obtains, 0～1 scope can certainly be handled by the mathematical operation that former state be used to carry out global motion.

Though used specific terms to describe the preferred embodiments of the present invention, should understand, under the situation of spirit that does not break away from accompanying claims or scope, can carry out various modifications and change.

Claims

1. image processing apparatus comprises:

Device for detecting motion vector is used for detecting each the motion vector among a plurality of of preliminary dimension that is arranged on image and formed by a plurality of pixels; And

The global motion calculation element; Be used to utilize at least one affine parameter wherein by affine transformation about the expansion of the function representation of the variable of the offset axis of said image; Employing is carried out the convergence mathematical operation by the said motion vector that said device for detecting motion vector detected, and puts on the global motion of the distortion of entire image with represents.

2. image processing apparatus according to claim 1; Wherein, In initial stage convergence mathematical operation through said global motion calculation element; Employing is by the affine transformation commonly used of said device for detecting motion vector detected said motion vector the carries out affine parameter that uses fixed coefficient, and after said initial stage convergence mathematical operation, begins to use the said convergence mathematical operation of the affine transformation of said expansion.

3. image processing apparatus according to claim 2, wherein, said global motion calculation element comprises:

Be used for using when utilizing said affine transformation commonly used to confirm said global motion used affine parameter to come each to calculate ideal movements vector to said through said convergence mathematical operation; Then, calculate said ideal movements vector and by the peak of the difference between the said said motion vector that said device for detecting motion vector detected and/or the device of mean value; And

Be used for when the peak of said difference and/or mean value become less than pre-set threshold, with the convergence mathematical operation that will use become the device of the convergence mathematical operation of the affine transformation of using said expansion from the convergence mathematical operation of using said affine transformation commonly used.

4. image processing apparatus according to claim 2, wherein, said device for detecting motion vector comprises:

The correlation value calculation device; Be used for confirming with the corresponding object block of the piece that is arranged on target picture and and be arranged on the correlation between a plurality of in the corresponding reference block of piece on the reference picture different with said target picture; Wherein, said a plurality of reference block is arranged in the hunting zone of a piece in the said object block;

Be used for confirming the peak of the said correlation that calculates by said correlation value calculation device and the peaked device of the said correlation except that said peak;

The object block device for detecting motion vector, the motion vector that is used to detect said object block is as the displacement with respect to said object block of the said reference block of the said peak that has calculated said correlation; And

Be used to calculate said peak and the difference between the said maximum of said correlation device as the reliability index of the motion vector of the said object block that is detected by said object block device for detecting motion vector;

Said global motion calculation element uses the motion vector that carries out weighting through the said reliability index of the motion vector of said object block, begins the convergence mathematical operation of said global motion.

5. image processing apparatus according to claim 2, wherein, said device for detecting motion vector comprises:

Said global motion calculation element, the said reliability index that comprises the motion vector that is used to use said object block are judged the device of reliability of the motion vector of said object block,

The said global motion calculation element only dependability motion vector that is judged as high said object block begins the initial stage convergence mathematical operation of said global motion.

6. image processing apparatus according to claim 1, wherein, said device for detecting motion vector comprises:

The correlation value calculation device; Be used for confirming with the corresponding object block of the piece that is arranged on target picture and and be arranged on the correlation between a plurality of in the corresponding reference block of piece on the reference picture different with said target picture; Wherein, said a plurality of reference block is arranged in the hunting zone in the piece of said object block;

7. image processing apparatus according to claim 1, wherein, said device for detecting motion vector comprises:

The said global motion calculation element only dependability motion vector that is judged as high object block begins the convergence mathematical operation of said global motion.

8. image processing apparatus according to claim 1; Wherein, Said global motion calculation element comprises and is used for using when confirming said global motion through said convergence mathematical operation used affine parameter to come to said each to calculate ideal movements vector, and calculates said ideal movements vector and by the device of the difference between the said motion vector of the piece that said checkout gear detected;

Use least square method and use the repeat number mathematical operations to carry out the said convergence mathematical operation of said global motion calculation element; Wherein, Said repeat number mathematical operations is carried out the mathematical operation of next circulation after removal shows the said motion vector of piece of peak of said difference.

9. image processing apparatus according to claim 8, wherein, said global motion calculation element comprises:

Be used for except that the said peak of said difference, also calculating the device of the mean value of said difference; And

The device of the completion of the convergence that is used for judging said repeat number mathematical operations through the said peak and the said mean value of said difference.

10. image processing apparatus according to claim 9, wherein, when remaining those motion vector numbers after removing through said repeat number mathematical operations become when being lower than predetermined value the computing of said global motion calculation element end global motion.

11. image processing apparatus according to claim 1; Wherein, Said global motion calculation element comprises and is used for using when confirming said global motion through said convergence mathematical operation used affine parameter to come to said each to calculate ideal movements vector, and calculates said ideal movements vector and by the device of the difference between the said motion vector of the piece that said checkout gear detected;

Use least square method and use the repeat number mathematical operations to carry out the said convergence mathematical operation of said global motion calculation element; Wherein, Said repeat number mathematical operations is carried out the mathematical operation of next circulation after removal shows the said motion vector of those pieces that said difference is higher than predetermined threshold value.

12. image processing apparatus according to claim 11, wherein, said global motion calculation element comprises:

Be used for except that the peak of said difference, also calculating the device of the mean value of said difference; And

13. image processing apparatus according to claim 12; Wherein, When remaining those motion vector numbers after removing through said repeat number mathematical operations become when being lower than predetermined value the computing of said global motion calculation element end global motion.

14. image processing apparatus according to claim 1 wherein, uses and carries out through the detection of said device for detecting motion vector to said motion vector through the downscaled images that downscaled images obtained.

15. image processing apparatus according to claim 14; Wherein, Said device for detecting motion vector use correlation in the downscaled images peak and through at correlation be said peak piece near the determined correlation of reference block carry out interpolation processing; And the result based on said interpolation processing detects each said motion vector of said

Said correlation represent be arranged on target picture in the corresponding object block of piece and and be arranged on the degree of correlation between a plurality of in the corresponding reference block of piece on the reference picture different with said target picture; Wherein, a plurality of reference blocks are arranged in the hunting zone of a piece in the said object block.

16. an image processing method comprises:

The motion vector detection step detects and to be arranged in the image and by each the motion vector among a plurality of of formed preliminary dimension of a plurality of pixels; And

The global motion calculation procedure; Utilize at least one affine parameter wherein by affine transformation about the expansion of the function representation of the variable of the offset axis of said image; Employing is carried out the convergence mathematical operation by the said motion vector that is detected in the said motion vector detection step, puts on the global motion of the distortion of entire image with represents.

17. image processing method according to claim 16; Wherein, In the convergence mathematical operation of the initial stage of said global motion calculation procedure; The motion vector that is employed in said of being detected in the said motion vector detection step is carried out the affine transformation commonly used of the affine parameter that uses fixed coefficient, and after said initial stage convergence mathematical operation, begins to use the said convergence mathematical operation of the affine transformation of said expansion.

18. image processing method according to claim 17, wherein, said global motion calculation procedure may further comprise the steps:

Use each during used affine parameter comes said when utilizing said affine transformation commonly used to confirm said global motion through said convergence mathematical operation is calculated ideal movements vector; Then, calculate said ideal movements vector and said said motion vector in the motion vector detection step, being detected between the peak and/or the mean value of difference; And

Be used for when the peak of said difference and/or mean value become less than pre-set threshold, with the convergence mathematical operation that will use become the convergence mathematical operation of the affine transformation of using said expansion from the convergence mathematical operation of using said affine transformation commonly used.