JPH10255049A - Image processing method using block matching - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the 'accurateness' of block matching. SOLUTION: A matching similarity operating part 14 and a matching intensity operating part 15 are provided, and matching similarity between a remarked block and ambient blocks and matching intensity that is a degree of matching value change to the change of a cutting position of a block from an image which is an object to be compared are calculated. They are taken into consideration when the weight to the ambient blocks are calculated, the weight is made small when the matching intensity of the ambient blocks is strong and the weight is made large when the matching intensity of them is conversely weak.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for comparing a plurality of images by block matching and detecting corresponding points on the images and motion vectors of the images, and a three-dimensional distance / shape using the results. The present invention relates to a method and apparatus for performing measurement, moving image compression / interpolation, and the like.

[0002]

2. Description of the Related Art Block matching is used for a problem of finding corresponding points between a plurality of slightly different images, for example, detection of a motion vector of a moving image, search for a corresponding point of a stereo image, and the like.

[0003] First, the structure of image data that forms the basis of image processing will be described. The image data handled here is two-dimensional image data in which pixel data is two-dimensionally arranged. The pixel data is data representing brightness at a corresponding position on an image, and one numerical value is used for a monochrome image, and three numerical values representing three primary colors are used for a color image.

Hereinafter, block matching will be described. In the block matching, first, reference image data and image data to be compared which are slightly different from the reference image data are input. For example, a frame of interest and a frame immediately before and after the frame of interest in the case of a moving image, and left and right images with different parallax in the case of a stereo image correspond to this.

FIG. 1 is a conceptual diagram for explaining an example of a pair of input images, that is, an image serving as a reference and an image to be compared, and cutting out a block therefrom. The reference image 1 and the comparison target image 2 include the subject 3. Reference numeral 4 denotes a block cut out from the reference image 1, and reference numeral 5 denotes a block corresponding to the block 4 on the image 2 to be compared.

FIG. 2 is a diagram showing an example of how blocks are cut out from an input image. First, one block 6 is cut out from the reference image 1. The block 6 includes a plurality of pixels 7, and a rectangular or square having a size of several to several tens of pixels (in the illustrated example, a square composed of 8 × 8 pixels) is used as the shape of the block.
Hereinafter, the extracted block 6 is referred to as a target block. Here, cutting out a block means making a partial copy of an image.

Next, a block having the same shape as that cut out from the reference image 1 is cut out from an arbitrary portion of the image 2 to be compared.
Is compared with the block 6 cut out from. For comparison, a method of calculating the square of the difference between pixel values at the same position in a block for all pixels in a block, taking the sum thereof, and assuming that the smaller the sum is, the more similar the blocks are to each other. Often used. Hereinafter, this sum is referred to as a matching value.

The block extraction from an arbitrary position and the comparison between the blocks are repeated, and the position where the most similar (minimized matching value) block is extracted is stored. The stored position is a desired position, that is, a position on the image 2 to be compared corresponding to a block cut out from the reference image 1. That is, in FIG. 1, the location of the block 5 corresponding to the target block 4 has been obtained. In the case of a moving image, the difference between the position of the block cut out from the reference image and the position of the corresponding block on the image to be compared is the motion vector, and the motion vector is detected. become.

In block matching, "accuracy" and "accuracy" become problems. Here, “accuracy” reflects the fluctuation around the true value, and “accuracy” reflects the mis-correspondence, that is, erroneously determining a part that is not originally a corresponding part as a corresponding part by mistake. I do. In general, as the size of a block increases, "accuracy" tends to increase and "accuracy" tends to decrease. Conversely, as the block size decreases, "accuracy" decreases and "accuracy" tends to improve. It is known that there is. Therefore, various methods have been proposed to achieve both "accuracy" and "accuracy" at a high level.

For example, IEICE Transactions, vol.J7
In 4-D-II, No.6, pp.669-677, the block size is dynamically selected from the surrounding conditions, and in places where "accuracy" is difficult to obtain, the block size is increased to improve "accuracy" In other places, the accuracy is maintained by reducing the size of the block.

Also, in IEICE Technical Report, IE95-40 (1995/7) and JP-A-3-217184, blocks around a block of interest are cut out, the matching values are weighted, and the matching value of the block of interest is weighted. Is added to Less than,
A block cut out from the periphery of the target block is called a peripheral block. In IEICE Technical Report, IE95-40 (1995/7), the same effect as changing the block size by changing this weight is obtained. JP-A-3-21
In Japanese Patent No. 7184, an effect similar to changing the weight by performing a non-linear operation on the matching value is obtained.

FIG. 3 is a diagram showing an example of how a block of interest and peripheral blocks are cut out from an input image. In the figure, reference numeral 1 denotes a reference image, 6 denotes a block of interest, 7 denotes one pixel, 8a to 8h denote peripheral blocks, and 3 denotes a subject.

For example, eight peripheral blocks 8a to 8h around the block of interest 6 are simultaneously cut out, and a total of nine blocks including the block of interest 6 are considered as one large block, and a matching value is calculated. However, when taking the sum, the weights set for each of the blocks 6, 8a to 8h are multiplied and then added to obtain a matching value. FIG.
Shows an example of a weighting coefficient.

If the weights of the peripheral blocks 8a to 8h are calculated in the same manner as that of the block 6 of interest, it is the same as when the size of the block is tripled (9 times in area). If the weights of the peripheral blocks 8a to 8h are all 0, the result is the same as when the calculation is performed for the target block 6 alone, and if the value is an intermediate value, the result is naturally an intermediate result. That is, by changing the weight, an effect similar to that of pseudo-changing the size of a block is produced. By dynamically determining the weight, the same effect as in IEICE Transactions, vol.J74-D-II, No.6, pp.669-677 is obtained.

Hereinafter, a method for dynamically determining the weight will be described. According to IEICE Technical Report IE95-40 (1995/7), when the matching situation between the block of interest and the neighboring block is similar, it is assumed that the motion in the image is a similar area, and the weight is increased. Otherwise, the weight is reduced. Here, there are several methods for calculating whether or not the matching situations are similar.

The following is an explanation of the method used in IEICE Technical Report, IE95-40 (1995/7). First, a matching value is calculated only once for a target block and a peripheral block using only a single block. At this time, the top N matching values from the smallest matching value and the position of the block cut out from the image to be compared at that time are stored. Calculate the number of the cutout positions of the blocks from the image to be compared which are common among the N blocks between the target block and any one of the peripheral blocks, and calculate the number between the target block and the peripheral block. , And used for weighting. This is performed for all the peripheral blocks, and the weight is determined for each.

[0017]

However, the methods of IEICE Technical Report, IE95-40 (1995/7) and JP-A-3-217184 pay attention only to the matching similarity when determining the weight. , A block with a small change in the matching value with respect to a change in the cutout position of the block from the image to be compared, which is poor in texture, etc., is adjacent to a location where the matching value is easily changed, such as the edge of an object, etc. Although the similarity of the matching is low due to the difference in the properties, the weight is small, but the weight may not be small enough.

Here, when the weight is not small enough,
If the target block is a block having a small change in the matching value with respect to the change in the cutout position of the block from the image to be compared, the change in the matching value is large with respect to the change in the cutout position of the block from the image to be compared. By adding the matching values of the surrounding blocks,
A matching value having a large change in the surrounding block has a greater effect on the result than a matching value having a small change in the block of interest, and the matching value of the block of interest corresponding to the block on the image to be compared , The matching value does not become the minimum, and as a result, a phenomenon that an erroneous correspondence occurs occurs.

The present invention has been made in view of such a problem of the related art, and has as its object to improve the “accuracy” of block matching. Also, the present invention
It is an object of the present invention to improve the quality of an image obtained by image processing using block matching and to improve the accuracy of measurement based on image processing using block matching.

[0020]

The above problem arises from the fact that it is not possible to cope with an excessively large degree of change in the matching value with respect to a change in the cutout position of a block from an image to be compared in a peripheral block. I have.
Hereinafter, the degree of the change in the matching value is referred to as the matching strength.

In the present invention, the matching strength is taken into account when calculating the weight, and when the matching strength of the peripheral block is strong, the weight is reduced so as not to have an unnecessarily large effect on the final result. If the matching strength of is low, the above problem is reduced by increasing the weight so that it can be used sufficiently and sufficiently to calculate the matching value.

Here, the simplest calculation method for calculating the matching strength is a calculation method in which the difference between the average of the entire matching values and the minimum value is used as the matching strength. The next conceivable method is a calculation method in which the difference between the average and the minimum value of the top M values of the matching values that are considered to have a large effect on the determination of the corresponding portion among the matching values is used as the matching strength. In addition, a calculation method in which the amount of change in the matching value around the minimum value of the matching value itself is used as the matching strength is also conceivable. In this case, for example, function approximation is performed around the minimum value of the matching value, and the parameter is used.

That is, according to the present invention, in an image processing method for detecting a motion vector and / or a corresponding point from two different images by a block matching method, a similarity and a matching strength of matching between a target block and neighboring blocks are determined. The final matching result is calculated by performing weighted addition of the matching results of the blocks near the block of interest using the weights determined in consideration.

Means for calculating the matching strength include:
The overall average of the matching, the higher average of the matching, or the amount of change around the matching can be used. Further, the present invention is characterized in that a corresponding point search of a stereo image is performed by using the image processing method, and a three-dimensional shape measurement and / or a distance measurement is performed by arithmetically processing the result.

Further, according to the present invention, a corresponding point search of a stereo image is performed by using the above-mentioned image processing method, the result is subjected to arithmetic processing, and an intermediate image of the stereo image is obtained from the stereo image by using means such as image interpolation and image transformation. One or a plurality of pseudo images, which are images from the viewpoint of, are generated.

Further, the present invention is characterized in that a motion vector of a moving image is detected by using the image processing method and the moving image is compressed. Further, the present invention performs motion vector detection of a moving image using the image processing method, and uses means such as image interpolation and image deformation from any two images in the moving image to obtain two images in a moving image. The method is characterized in that one or a plurality of pseudo images serving as intermediate images of the above images are generated.

As described above, in the present invention, two images are input, one of which is set as the reference side image, and the other is set as the comparison side image. Cut out a block from the reference side image. This is called a target block. Further, a block is cut out from the vicinity of the block of interest. This is called a peripheral block. Blocks are similarly cut out from the image to be compared, and the matching value is calculated between single blocks. Here, the matching value is a value indicating how similar a block cut out from the reference side image is to a block cut out from the comparison side image.

Next, the matching similarity between the target block and the surrounding blocks is calculated from the calculated matching values.
The matching similarity indicates how similar the matching tendency of the target block and the surrounding block is. Also,
The matching strength of the target block and the surrounding blocks is calculated from the calculated matching value. The matching strength indicates the strength of matching of each block, that is, the degree of change in the matching value.

A weight is determined for each peripheral block from the calculated matching similarity and matching strength. The weight is multiplied and the matching value is calculated again, and the position where the matching value is minimized is determined as the position to be obtained. By considering the matching strength and the matching similarity in determining the weight, higher “accuracy” than the conventional method is obtained.

[0030]

Embodiments of the present invention will be described below with reference to the drawings. [Embodiment 1] FIG. 5 is a system configuration diagram of an image processing apparatus according to an embodiment of the present invention. This image processing apparatus performs three-dimensional shape / distance measurement using a stereo image, and obtains three-dimensional shape / distance data by capturing an object with two cameras, inputting the stereo image, and processing the image using a computer main body. . The obtained data is displayed on the display device and simultaneously recorded in the external storage device.

The image processing apparatus main body 10 includes a CPU (central processing unit) 11, a memory 12, a matching value calculation unit 13,
Matching similarity calculator 14, matching strength calculator 1
5, auxiliary storage device 16, image input interface unit 1
7, an external storage device interface unit 18 and a user interface unit 19 are provided. The cameras 21 and 22 for photographing the measurement target are provided to the image input interface unit 17,
The external storage device 23 is an external storage device interface unit 1
8, a display device 24, a keyboard 25, and a mouse 26 are connected to the user interface unit 19, respectively.

FIG. 6 is a detailed diagram of the matching value calculator 13. The matching value calculation unit 13 includes a block 1 storage unit 31 and a block 2 storage unit 32 that store a reference block and a comparison target block, and a subtraction circuit 33 that calculates a difference between pixels at the same position in the two blocks. , A sum-of-products circuit 34 which calculates the square of the subtracted value and sums the sum of all the pixels in the block, and a matching value storage unit 3 which stores the calculated value of the sum-of-products circuit 34 as a matching value and a value
Consists of five.

FIG. 7 is a detailed diagram of the matching similarity calculating section 14. The matching similarity calculation unit 14 detects the block-of-interest matching value storage unit 41, the peripheral block matching value storage unit 42, the rearranging circuits 43 and 44 that arrange the matching values in ascending order, and detects a common block between the target block and the peripheral block. Common part detection circuit 4
5, a counter 46 for counting the number of common parts, a division circuit 47 for normalizing the count value of the counter 46, and a matching similarity storage unit 48 for storing the output of the division circuit 47 as the matching similarity.

FIG. 8 is a detailed diagram of the matching strength calculator 15. The matching strength calculation unit 15 includes a matching value storage unit 51, a product-sum circuit 52 and a division circuit 53 for calculating the average, a minimum value storage unit 54 for calculating and storing the minimum value of the matching value, and an output of the division circuit 53. Circuit 5 for calculating the difference between the minimum value and the minimum value stored in minimum value storage unit 54
5, a matching strength storage unit 56 for storing the difference as a matching strength.

In this embodiment, two left and right cameras 2
Although 1 and 22 are used, the positional relationship and the number of cameras are not limited to the examples shown here. For example, Journal of the Institute of Image Electronics Engineers of Japan, Vol. 20, No. 4, 380-38
As described on page 6, cameras may be arranged vertically. Further, the camera and the photographing target are not necessarily fixed and do not move, and other than the camera may be used as the image input means.

Next, an image processing method will be described with reference to the flowchart of FIG. First, in step 11 of FIG. 9, stereo image data is input from the left and right cameras 21 and 22 and stored in the memory 12 of the image processing device 10. Here, two images taken by the two cameras 21 and 22
One of the image data is set as a reference side, and the other is set as a comparison target side.

Next, in step 12, the memory 12
An 8 × 8 pixel block 6 (FIG. 3) is read from an arbitrary position of the reference side image stored above and stored (copied) in a calculation buffer area at another position in the memory 12. Hereinafter, this operation is referred to as clipping. The block cut out here is called a block of interest.

Next, in step 13, from the periphery of the block of interest 6, the entire shape of the cut-out block becomes 3
8 × 8 pixel blocks 8a to 8 ×
8h (FIG. 3) is cut out. These eight blocks 8a
８8h are called peripheral blocks.

Here, the size of the block is set to 8 × 8 pixels, and the description will be given by the method of cutting out the peripheral block as shown in FIG. 3, but the size of the block, the positional relationship between the target block and the peripheral block, and the peripheral block The number and the like are not limited to the example shown here, and can be variously changed.

Next, in step 14, 3 × 3 blocks are cut out of blocks having the same shape as those cut out from the reference side from the image data on the comparison side, that is, blocks of 8 × 8 pixels.

Subsequently, in step 15, the matching value between the corresponding blocks on the reference side and the comparison side is calculated by the matching value calculation unit 13 shown in detail in FIG. That is, the information of the cut-out block is transferred to the block 1 storage unit 31 and the block 2 storage unit 32 inside the matching value calculation unit 13, and the subtraction circuit 33 and the product-sum circuit 34 use a certain pixel of a certain block on the reference side. Against
The square of the difference from the pixel at the same position in the corresponding block on the comparison target side is calculated, and the sum is calculated for all the pixels in the block. The obtained sum is stored in the matching value storage unit 35.
After that, the data is transferred to the memory 12 (FIG. 5) of the image processing apparatus 10. This is performed for all the blocks, and the obtained nine matching values are stored in the memory 12 together with the cutout positions of the blocks from the comparison target side.

Steps 14 to 15 are repeatedly executed for a certain range on the comparison target side. Usually, this range is used within a predetermined distance set in advance around the same position as the cut-out position on the reference side. If it is determined in step 16 that the comparison target image has been cut out within the range and the calculation of the matching value has been completed, the process proceeds to step 17.

In step 17, the peripheral blocks 8 extracted from all the reference side images are extracted from the extraction positions and the matching values calculated in steps 14 to 16.
With respect to a to 8h, the similarity of matching with the block of interest 7 is calculated.

FIG. 10 shows the details of step 17. First, the stored data is transferred to the target block matching value storage unit 41 and the peripheral block matching value storage unit 42 inside the matching similarity calculation unit 14 shown in detail in FIG. Next, in step 31, the rearranging circuit 4
According to 3, 44, the blocks are rearranged in ascending order of the matching value for each block. Next, in step 32, the upper N matching values and their cut-out positions are extracted from the smaller matching values using a preset constant N. In step 33, the common part detection circuit 45 and the common block detection circuit 45 determine how many of the upper N cutout positions extracted from each of the target block 6 and the peripheral blocks 8a to 8h whose weights are to be determined are common. Calculate at the counter 46. The minimum is 0 and the maximum is N because all are common. In step 34, this number is divided by N by the dividing circuit 47 and normalized to obtain a similarity of matching. This matching similarity is stored in the matching similarity storage unit 48 and then transferred to the memory 12 of the image processing apparatus 10 and stored.

The operation of step 17 in FIG. 9 is performed on all the peripheral blocks 8a to 8h, and the matching similarity of all the peripheral blocks with the target block is calculated. .

In step 19, the matching strength between the block of interest 6 and the peripheral blocks 8a to 8h is determined from the cutout positions and the matching values calculated in steps 14 to 16, and the matching strength calculation unit 1 shown in FIG.
Calculate at 5. FIG. 11 is a diagram showing details of step 19.

In step 19, first, the matching value of the block whose matching strength is to be calculated is transferred to the matching value storage unit 51. Next, in step 41 of FIG. 11, the average is calculated by the product-sum circuit 52 and the division circuit 53. Further, the minimum value of the matching value is calculated and stored in the minimum value storage unit 54. Next, in step 42, the difference between the average and the minimum value of the matching values is calculated by the subtraction circuit 55. This difference is used as the matching strength, stored in the matching strength storage unit 56, and transferred to and stored in the memory 12 of the image processing apparatus 10.

The operation of step 19 is performed on the block of interest 6 and all the peripheral blocks 8a to 8h to calculate the matching strength. When that is done, step 20
To step 21.

In step 21, the matching similarity calculated in steps 17 to 18 and step 1
From the matching strength calculated in step 20 from step 9,
The weight is calculated as shown in FIG. In this example, the weight uses the product of the matching similarity and the reciprocal of the matching strength. However, since there is no matching similarity for the block of interest 6, the matching similarity is substituted by an arbitrary constant. In the example of the figure, 2 is used for this constant. The reason why this constant is set to a value exceeding the maximum value 1 of the matching similarity is to improve the accuracy by increasing the specific gravity of the block of interest. The weight can be calculated by using a function having a positive correlation with both the product of the matching similarity and the reciprocal of the matching strength in accordance with the purpose.

Next, in step 22, step 2
Using the weight calculated in step 1, a matching value is calculated for the entire block cut out from the reference side pixel. In this example, since the block of 8 × 8 pixels is a 3 × 3 block, the calculation is substantially the same as the calculation of the matching value in the block of 24 × 24 pixels except for multiplying the weight. However, we have already calculated the matching value and position for each block,
Since it is stored in the memory 12, the matching value can be calculated only by multiplying the matching value at a certain position by the weight of the block and calculating the sum of the target block 6 and all the peripheral blocks 8a to 8h. The matching value and the position thus calculated are stored in the memory 12 again.

The operation in step 22 is repeatedly executed for the same range as in step 16. When this operation ends, the process proceeds from the step 23 to a step 24. Step 2
In step 4, the smallest one of the matching values calculated in steps 22 to 23 is calculated. The position of the smallest one is the position of the corresponding block in the image on the comparison target side with respect to the target block, for example, block 4 in FIG.
Is the position of the block 5 corresponding to.

The operations from step 12 to step 24 are repeatedly executed for a certain range of the reference side image. Usually, the entire reference side image is used for this range. As a result, a portion corresponding to the comparison target side image is obtained for the entire reference side image. When this operation ends, the process advances from the step 25 to the step 26.

In step 26, if the corresponding portion of the image to be compared has been obtained for the entire image on the reference side, the position of the arbitrary portion on the reference image is determined along with the position information of the camera.
The dimensional position can be determined by triangulation. As this calculation method, a widely known one can be used.

According to this embodiment, the "accuracy" of the block matching is improved, and erroneous correspondence is reduced especially in the vicinity of a boundary region between the object and the background where the motion is different and the texture is poor compared to the conventional method. can do. Also, 3
The accuracy of dimensional shape / distance measurement can be improved.

[Second Embodiment] The matching strength calculation section 15 of the image processing apparatus 10 is configured as shown in FIG. 12, and the matching strength is calculated by the processing shown in FIG. The rest is the same as in the first embodiment, and therefore only the calculation of the matching strength in step 19 in FIG. 9 will be described here.

As shown in FIG. 12, the matching strength calculation section 15 includes a matching value storage section 61, a rearrangement circuit 62 for rearranging the stored matching values according to their magnitudes,
A product-sum circuit 63 and a division circuit 64 for calculating the average of the sorted matching values, a minimum value storage unit 65 for calculating and storing the minimum value of the sorted matching values, an output and a minimum value of the division circuit 64 The subtraction circuit 66 calculates a difference from the minimum value stored in the storage unit 65, and a matching strength storage unit 67 stores the difference as a matching strength.

Next, the calculation of the matching strength will be described. In step 19 in FIG. 9, first, the matching value of the block whose matching strength is to be calculated is transferred to the matching value storage unit 61. Next, in step 51 of FIG. 13, the matching values of the blocks to be calculated as the matching strength are rearranged in the rearrangement circuit 62 in ascending order. Next, in step 52, the average of the M matching strengths from the smallest one is calculated by the product-sum circuit 63 and the division circuit 64.
Is calculated by Here, M is an arbitrary constant given in advance. Further, the minimum value of the matching value is calculated and stored in the minimum value storage unit 65. Next, in step 53,
The difference between the average and the minimum value of the matching value is subtracted by a subtraction circuit 66.
Is calculated by This difference is used as the matching strength, stored in the matching strength storage unit 67, and transferred to and stored in the memory 12 of the image processing apparatus 10. According to this embodiment, higher “accuracy” of block matching can be realized by adding a simple circuit.

[Third Embodiment] The matching strength calculation unit 15 of the image processing apparatus 10 is configured as shown in FIG. 14, and the matching strength is calculated by the processing shown in FIG. The rest is the same as in the first embodiment, and therefore only the calculation of the matching strength in step 19 in FIG. 9 will be described here.

As shown in FIG. 14, the matching strength calculation section 15 includes a matching value storage section 71, a neighborhood matching value storage section 72, a rearrangement circuit 62 for rearranging the stored matching values according to the magnitude thereof, and a matching circuit 62 for obtaining the minimum value. It comprises a minimum value storage circuit 74 for storing the minimum value of the values, a parameter estimator 75, and a matching strength storage 76.

Next, the calculation of the matching strength will be described. In step 19 in FIG. 9, first, the matching value of the block whose matching strength is to be calculated is transferred to the matching value storage unit 71. Next, in step 61 of FIG. 15, the minimum value of the matching value of the block whose matching strength is to be calculated is obtained by the reordering circuit 73, stored in the minimum value storage unit 74, and the minimum value of the block cut out from the comparison target side image at that time. Find the position. Next, in step 62, the matching value when the neighboring block is cut out is extracted from the matching value storage unit 71 and stored in the neighborhood matching value storage unit 72. Next, step 6
In 3, the parameter estimating unit 75 estimates the amount of change in the matching value from these values.

In this example, the minimum value and the three values on both sides are applied to a quadratic function of the following [Equation 1] to obtain a parameter a of the function.
Is used as a value representing the amount of change. That is, this parameter a is used as the matching strength. Then, the matching strength is stored in the matching strength storage unit 76, and is transferred to and stored in the memory 12 of the image processing apparatus 10. In [Equation 1], y is a matching value, and x is a variable representing the position of each block. Since three equations are given for the three variables a, b, and c, the values of a, b, and c can be obtained.

[0062]

Y = ax ² + bx + c According to this embodiment, the circuit is slightly complicated, but higher “accuracy” of block matching can be achieved.

[Embodiment 4] An embodiment in which an intermediate image is formed from two images will be described. As in the first to third embodiments, the cameras 21 and 22 are transmitted from the image input interface unit 17 of the image processing layer value 10 shown in FIG.
Input two images from, and find corresponding points between the images.
Next, step 26 in the first to third embodiments (FIG. 9)
Instead of performing the process of calculating the three-dimensional position, an intermediate image between the two images is generated using the obtained corresponding point information.

Various methods are known for generating an intermediate image between two images by image interpolation and image deformation using the obtained corresponding point information. For example, the following method can be used. it can.

First, neighboring points on the reference side image are connected by a line to form a net. Based on the corresponding point information, it is calculated how this network will be deformed on the image to be compared. The intermediate state of the deformation is also calculated by multiplying the movement vector to the corresponding point by a coefficient α.
With respect to the shape of the net in the intermediate state, an image is generated by deforming the corresponding portion of the net on the reference side image and the comparison target side image along the shape of the net, and a deformed image generated from the reference side image is generated. On the other hand, a single intermediate image is generated by multiplying the modified image generated from the image to be compared by (1-α) and multiplying by (1−α).

In the generated intermediate image, if there is a portion processed by an erroneous vector, a sense of incongruity occurs in that portion. In this embodiment, “accuracy” is compared with the conventional method.
By applying the improved block matching technique, the visual quality of the resulting image is improved.

[Embodiment 5] FIG. 16 is a system configuration diagram of an image processing apparatus according to another embodiment of the present invention. This image processing apparatus performs moving image compression. A moving image from a moving image source is input through a moving image input block and processed by a computer main body to obtain compressed image data. The obtained data is recorded in the external storage device through the external storage device interface unit.

The image processing apparatus main body 10 includes a CPU (central processing unit) 11, a memory 12, a matching value calculation unit 13,
Matching similarity calculator 14, matching strength calculator 1
5, an auxiliary storage device 16, a moving image input interface unit 81, an external storage device interface unit 18, and a user interface unit 19. A moving image source 82 such as a camera or a video is provided to the moving image input interface unit 81,
The external storage device 23 is an external storage device interface unit 1
8, a display device 24, a keyboard 25, and a mouse 26 are connected to the user interface unit 19, respectively. As the matching value calculation unit 13, matching similarity calculation unit 14, and matching strength calculation unit 15, those described above can be used.

The moving image input interface unit 81 inputs two of the images constituting the moving image from the moving image source 82. Normally, two continuous or two temporally continuous images are input on a moving image. One of the two images is the reference side. The same processing as in steps 12 to 25 (FIG. 9) of the above embodiment is performed on the two input images. As a result, a motion vector between the two images is obtained for the entire image on the reference side. Various methods are known as an algorithm for compressing a moving image using the motion vector. For example, the following method can be used.

Using the motion vector obtained from the two images, a block having the same size as that obtained when the motion vector was obtained from the reference side image is taken out and moved in parallel according to the motion vector. This operation is performed on the entire reference side image, and a single image is generated by superimposing. next,
The difference between this image and the image to be compared is taken. This difference is called a prediction error image, and has the characteristic that it can be compressed at a higher compression ratio even with the same compression means as compared to the image to be compared. The same compression as that of a normal still image is performed on this difference.

If there is a part processed by an erroneous vector in the image compression, the reproduced image after decompression has a strange feeling. In this embodiment, the visual quality of the resulting image can be improved by using the image processing technique with improved “accuracy” of block matching according to the present invention for compression of a moving image or the like. it can.

[Embodiment 6] An image processing apparatus according to another embodiment of the present invention will be described. In this embodiment, the moving image interpolation is performed using the system configuration shown in FIG. 16 as in the fifth embodiment. A moving image from a moving image source is input through a moving image input block, processed by a computer main body, and interpolated image data is obtained. The obtained data is recorded in the external storage device through the external storage device interface unit.

A moving image input interface unit 81 inputs two of a group of images constituting a moving image from a moving image source such as a camera or a video to obtain a motion vector.
An intermediate image of two images in the moving image is generated from the obtained motion vector by image interpolation and image deformation. Various methods are known for this method. For example, the following methods can be used.

Using the motion vector obtained from the two images, a block having the same size as that obtained when the motion vector is obtained from the reference side image is taken out, and is moved in parallel according to a vector obtained by multiplying the motion vector by a constant β. This operation is performed on the entire reference side image, and a single image is generated by superimposing. The same operation is performed on the comparison target image. However, the motion vector is multiplied by (1−β). To the two images obtained in this way, the image obtained from the reference side image is multiplied by β, and the image obtained from the comparison target side image is multiplied by (1−β) and added. Combine images. This is the intermediate image.

If there is a portion processed by an erroneous vector, a sense of incongruity occurs in the generated intermediate image. According to the present embodiment, by applying an image processing method that improves the “accuracy” of block matching according to the present invention to the generation of an interpolation / intermediate image such as a moving image, the visual quality of the resulting image can be improved. Quality is improved.

[0076]

According to the present invention, the "accuracy" of block matching can be improved, and erroneous correspondence in the vicinity of a boundary region between the object and the background having a different texture and poor texture can be reduced. In addition, by using the block matching with improved “accuracy” in this way, it is possible to improve measurement accuracy and visual quality of an image.

[Brief description of the drawings]

FIG. 1 is a conceptual diagram for explaining an example of a pair of input images, cutting out blocks from the input images, and corresponding portions.

FIG. 2 is a diagram showing an example of a state in which a block is cut out from an input image.

FIG. 3 is a diagram illustrating an example of a state in which a target block and peripheral blocks are cut out from an input image.

FIG. 4 is an explanatory diagram of a weighting coefficient.

FIG. 5 is a system configuration diagram of an image processing apparatus according to an embodiment of the present invention.

FIG. 6 is a detailed diagram of a matching value calculation unit.

FIG. 7 is a detailed diagram of a matching similarity calculation unit.

FIG. 8 is a detailed diagram of a matching strength calculation unit.

FIG. 9 is a diagram showing a flowchart of a process.

FIG. 10 is a diagram showing an example of a processing flow of matching similarity calculation.

FIG. 11 is a diagram showing an example of a processing flow of matching strength calculation.

FIG. 12 is a detailed view of another example of the matching strength calculation unit.

FIG. 13 is a diagram showing another example of the processing flow of the matching strength calculation.

FIG. 14 is a detailed view of another example of the matching strength calculation unit.

FIG. 15 is a diagram showing another example of the processing flow of the matching strength calculation.

FIG. 16 is a system configuration diagram of an image processing apparatus according to another embodiment of the present invention.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Reference image, 2 ... Comparative image, 3 ... Subject, 4 ... Block cut out from the reference image, 5 ... Block corresponding to the block on the image to be compared, 6 ... Target block, 7 ... Pixel , 8a-8h ... peripheral blocks, 10 ...
Image processing device, 11 CPU (central processing unit), 12
Memory, 13 matching value calculation unit, 14 matching similarity calculation unit, 15 matching strength calculation unit, 16 auxiliary storage device, 17 image input interface unit, 18
... external storage device interface unit, 19 ... user interface unit, 21, 22 ... camera, 23 ... external storage device, 24 ... display device, 25 ... keyboard, 26 ... mouse, 81 ... moving image input interface unit, 82 ... moving image Source 31: Block 1 storage unit 32: Block 2 storage unit 33: Subtraction circuit 34: Product-sum circuit 35: Matching value storage unit 41: Target block matching value storage unit 42: Peripheral block matching value storage Part, 43, 4
4 ... rearrangement circuit, 45 ... common part detection circuit, 46 ... counter, 47 ... division circuit, 48 ... matching similarity storage unit, 51 ... matching value storage unit, 52 ... product-sum circuit, 5
3: division circuit, 54: minimum value storage unit, 55: subtraction circuit,
56: matching strength storage unit, 61: matching value storage unit, 62: rearrangement circuit, 63: product-sum circuit, 64: division circuit, 65: minimum value storage unit, 66: subtraction circuit, 67: matching strength storage unit, 71 ... matching value storage unit, 72
... Neighborhood matching value storage unit 73, rearrangement circuit 74
... Minimum value storage unit, 75 ... Parameter estimation unit, 76 ... Matching strength storage unit

Claims (8)

[Claims] 1. An image processing method for detecting a motion vector and / or a corresponding point from two mutually different images by a block matching method, wherein the determination is made in consideration of a similarity and a matching strength of matching between a target block and a nearby block. An image processing method, wherein a final matching result is calculated by performing weighted addition of a matching result of a block near a block of interest using the calculated weight. 2. The image processing method according to claim 1, wherein
An image processing method, wherein an overall average of matching is used as means for calculating the matching strength. 3. The image processing method according to claim 1, wherein
An image processing method characterized by using a higher average of matching as a means for calculating the matching strength. 4. The image processing method according to claim 1, wherein
An image processing method, wherein a change amount around matching is used as means for calculating the matching strength. 5. A three-dimensional shape measurement and / or a distance measurement by performing a corresponding point search of a stereo image using the image processing method according to any one of claims 1 to 4, and processing the result. Performing the method. 6. A method for searching a corresponding point of a stereo image by using the image processing method according to claim 1, calculating a result of the search, and performing image interpolation, image transformation, and the like from the stereo image. An image processing method for generating one or a plurality of pseudo images as images from an intermediate viewpoint of a stereo image using the means. 7. An image processing method comprising compressing a moving image by detecting a motion vector of the moving image using the image processing method according to claim 1. 8. A motion vector of a moving image is detected using the image processing method according to any one of claims 1 to 4, and image interpolation, image deformation, and the like are performed from any two images in the moving image. An image processing method comprising: generating one or a plurality of pseudo images serving as intermediate images between two images in a moving image by using the means.