JP2010278931A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a block matching technology effective for improving the resolution of a moving image. <P>SOLUTION: An image processing apparatus comprises: a block matching section which performs block matching between a plurality of frames to determine a corresponding point of a block between the frames; a threshold determining section which compares a matching error outputted from the block matching section with a threshold to output a corresponding point with integral precision; a decimal precision aligning section for creating a corresponding point with decimal precision when the corresponding point with integral precision is inputted; a flatness analyzing section in which the flatness of an image of the block is calculated and if the flatness is high, the threshold is controlled to become smaller than the case that the flatness is low; and an interpolation image generating section for generating an interpolation image having a higher resolution from the inputted frame signal based on a plurality of corresponding points with decimal precision. The interpolation image generating section generates the interpolation image using the corresponding points with decimal precision when it is determined that the matching error output is smaller than or equal to the threshold. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that performs high resolution processing.

  When displaying a low-resolution moving image on a high-resolution display, there is a problem that the image is blurred in the conventional enlargement by linear interpolation or cubic interpolation. Therefore, a high resolution technique using a plurality of pieces of frame information is useful. As described in Patent Document 1, in order to obtain a high-resolution image using a plurality of pieces of frame information, it is necessary to align pixels between frames. One method for aligning pixels is to perform pixel block matching between frames.

  However, conventionally, a block matching technique effective in increasing the resolution of moving images has not been disclosed.

JP 2009-81734 A

  An object of the present invention is to provide a block matching technique effective in increasing the resolution of a moving image.

  In order to solve the above-described problems, an image processing apparatus according to the present invention includes a block matching unit that performs block matching between a plurality of frames and determines corresponding points of the blocks between frames, and a matching error output of the block matching unit. A threshold value determination unit that compares values and outputs corresponding points of integer precision; a decimal precision alignment unit that generates corresponding points of decimal precision when the corresponding points of integer precision are input; and an image of the block Based on the flatness analysis control unit for controlling the threshold value to be lower than when the flatness is low and the corresponding points of the plurality of decimal precision, An interpolated image generating unit that generates a higher resolution interpolated image from the input frame signal, and the interpolated image generating unit has the matching error output equal to or lower than the threshold value. If it is determined that the interpolated image is generated using the decimal precision corresponding point, and the matching error output is not determined to be equal to or less than the threshold value, the decimal precision corresponding point is not used. The interpolation image is generated only from the pixels of the frame.

  According to the present invention, a block matching technique effective in increasing the resolution of a moving image can be obtained.

The schematic diagram which shows the flame | frame structure of a moving image. Explanatory drawing which shows the high resolution process using the some frame information of one Embodiment of this invention. Explanatory drawing which shows the interpolation pixel production | generation of the embodiment. The block diagram of the high resolution image processing apparatus of the embodiment. Explanatory drawing which shows the corresponding point search by the block matching used for the embodiment. Explanatory drawing which shows the pixel block of the embodiment. The block diagram of the corresponding point search part 43 of the embodiment. Explanatory drawing which shows the flatness detection method used for the embodiment. Explanatory drawing which shows another example of the flatness detection method used for the embodiment. Explanatory drawing which shows the setting method of the threshold value used for the embodiment. The flowchart of the process in the corresponding point search part 43 of the embodiment. The block diagram of the conventional corresponding point search part. Explanatory drawing which shows an example of the matching error in an image with low flatness. Explanatory drawing which shows an example of the matching error in an image with high flatness.

Embodiments according to the present invention will be described with reference to the drawings.
FIG. 1 is a schematic diagram showing a frame configuration of a moving image. With respect to the video of the car at the time Ta in the sequence of continuous frames of the moving image, the car progresses such that Tb after 7 frames and Tc after 7 frames.

  FIG. 2 is an explanatory diagram illustrating high resolution processing using a plurality of frame information according to the embodiment. FIG. 2 shows a mechanism for generating an image having a higher resolution than the input image using a plurality of pieces of frame information in the moving image as shown in FIG.

  Now, assuming that pixels are arranged at the intersections of the solid lines shaded in FIG. 2, an interpolation pixel located at the intersection of the dotted lines of the Nth frame is generated and converted into an image having twice the vertical and horizontal pixel counts. Assume a case. When estimating the interpolated pixels, using only the pixels in the Nth frame (indicated by black circles) increases the resolution to restore the original image of the subject such as so-called super-resolution due to lack of information. I can't do it. Therefore, by bringing pixel information indicated by triangles and diamonds from frames before and after the Nth frame, it is possible to estimate an interpolation pixel using more information as shown in FIG. And higher resolution is possible. In FIG. 2, information for each frame before and after is used, but the accuracy of interpolation pixel estimation can be increased by using more frame information.

  FIG. 3 is an explanatory diagram illustrating interpolation pixel generation according to the embodiment. FIG. 3 corresponds to the second region from the left and the second region from the top surrounded by the solid line in FIG. That is, the above pixel information is used to generate an interpolated pixel indicated by a white circle.

FIG. 4 is a configuration diagram of the high-resolution image processing apparatus according to the embodiment.
The high resolution image processing apparatus includes a frame memory 41, a frame memory 42, a corresponding point search unit 43, and an interpolated image generation unit 44.
The frame memory 41 delays the input signal by one frame, and the frame memory 42 delays the input signal by another one frame (a total of two frames).
The corresponding point search unit 43 generates corresponding point information from the input signal, the output signal of the frame memory 41, and the output signal of the frame memory 42. Specific examples of the corresponding point information include the position and luminance of the corresponding point, and further the hue. The expression of the position information of the corresponding points may be a vector.

The interpolation image generation unit 44 generates an interpolation image from the output signal of the frame memory 41 based on the output of the corresponding point search unit 43.
The panel 45 displays the signal generated by the interpolation image generation unit 44. In addition, the interpolation frame may be added to the 60 Hz output of the interpolation image generation unit 44 using a conventional technique to form a 120 Hz image and then led to the panel 45.

  As shown in FIG. 2, in order to bring the pixel information of the (N-1) th frame to the Nth frame, the coordinates of each pixel in the (N-1) th frame correspond to the coordinates of the Nth frame. And the corresponding point search unit 43 in FIG. 4 performs this process.

  FIG. 12 is a configuration diagram of a corresponding point search unit using a conventional technique called block matching. The corresponding point search unit includes a block matching unit 81 that performs block matching from the 2-frame delay signal and the 1-frame delay signal, and a threshold determination unit 82 that performs threshold determination described later on the output of the block matching unit 81. A decimal precision alignment unit 83 that performs decimal precision alignment based on the output of the threshold value determination unit 82; The corresponding point search unit includes a block matching unit 84 that performs block matching from the one-frame delay signal and the input signal, a threshold determination unit 85 that performs threshold determination described later on the output of the block matching unit 84, and a threshold value. A decimal precision alignment section 86 that performs decimal precision alignment from the output of the determination section 85. FIG. 5 is an explanatory diagram showing a state of corresponding point search by block matching. Now, when it is examined at which coordinate in the Nth frame a certain pixel (target pixel) in the (N-1) th frame is located, a pixel block (3 × 3 pixels or 5 × 5 pixels) centered on the target pixel is considered. The pixel block having a high correlation with the pixel etc. is searched from the Nth frame.

FIG. 6 is an explanatory diagram showing a pixel block and represents a 5 × 5 pixel block.
Here, a sum of absolute values of differences (SAD) or a sum of squares of differences (SSD) is often used as a function for evaluating the correlation between pixel blocks. If there is a 5 × 5 pixel block in the (N−1) th frame as shown in FIG. 6A and a 5 × 5 pixel block in the Nth frame as shown in FIG. 6B, SAD and SSD are Each is represented by the following formula. However, Aij and Bij indicate luminance values, respectively.

  As shown in FIG. 5, the position where the SAD of the pixel block is the smallest in the Nth frame is the corresponding point obtained with integer precision. After block matching, the threshold value determination unit 82 or 85 compares the SAD value with the threshold value as shown in FIG. When SAD is larger than the threshold value, it is determined that the reliability is low, and the target pixel is excluded from the corresponding points. After the threshold determination, the position is further aligned with decimal precision. For the decimal precision alignment, a method of obtaining from the SAD value using a fitting function is widely used. An example of the fitting function is an equiangular straight line.

  The above processing is a conventional method for increasing the resolution of a moving image using a plurality of frame information. However, if the SAD value is simply compared with the threshold value as shown in FIG. The problem of misrecognizing Examples thereof are shown in FIGS. FIG. 13 is an explanatory diagram illustrating an example of a matching error in an image with low flatness. FIG. 14 is an explanatory diagram illustrating an example of a matching error in an image with high flatness.

  If there is an object having a luminance distribution with low flatness as shown in FIG. 13A, the luminance data sampled at the position indicated by the dotted line is as shown in FIG. 13C. Further, luminance data obtained by sampling the same object at a position indicated by a dotted line in FIG. 13B is as shown in FIG. If there is data as shown in FIG. 13C in one of the two image frames and data as shown in FIG. 13D in the other, and these block matchings are performed, even the most highly correlated part is shown in FIG. It can be seen that a matching error as shown in e) occurs. The matching error is obtained as the minimum value of the SAD value at the corresponding point obtained with the integer precision.

  On the other hand, if there is an object having a highly flat luminance distribution as shown in FIG. 14A, luminance data sampled at a position indicated by a dotted line is as shown in FIG. Also, assuming that there is a different object with high flatness as shown in FIG. 14B, the luminance data sampled at the position indicated by the dotted line is as shown in FIG. When block matching of these luminance data is performed, the matching error is as shown in FIG. 14 (e), and it can be seen that the matching error is not so large as compared with FIG. 13 (e).

  As described above, the matching error is large even in the same object in the pattern portion with low flatness, and the matching error is small in the different object in the pattern portion with high flatness. In such a situation, when reliability is determined with the same threshold value, correct corresponding points are not used for pattern portions with low flatness, and incorrect corresponding points are used for pattern portions with high flatness. There are many problems that can occur. When an incorrect corresponding point is used in a pattern portion with high flatness, dot-like noise or the like is generated, and the image quality of the high-resolution image is degraded.

FIG. 7 is a configuration diagram of the corresponding point search unit 43 of the embodiment.
The corresponding point search unit 43 includes a block matching unit 71, a threshold value determination unit 72, a decimal accuracy alignment unit 73, a flatness analysis control unit 77 and a block matching unit 74, a threshold value determination unit 75, and a decimal accuracy alignment unit. 76 and a flatness analysis control unit 78.

  The block matching unit 71 performs block matching from the 2-frame delay signal and the 1-frame delay signal, calculates the above-described matching error, and outputs it to the threshold value determination unit 72. The threshold determination unit 72 performs threshold determination described later on the output of the block matching unit 71. The decimal precision alignment unit 73 performs decimal precision alignment from the output of the threshold value determination unit 72. The flatness analysis control unit 77 analyzes the flatness from the 2-frame delay signal and the 1-frame delay signal, and transmits a threshold value calculated based on the flatness to the threshold value determination unit 72.

  The block matching unit 74 performs block matching from the one-frame delay signal and the input signal, calculates the above-described matching error, and outputs it to the threshold value determination unit 75. The threshold determination unit 75 performs threshold determination described later on the output of the block matching unit 74. The decimal precision alignment unit 76 performs decimal precision alignment from the output of the threshold value determination unit 75. The flatness analysis control unit 78 analyzes the flatness from the one-frame delay signal and the input signal and transmits a threshold value calculated based on the flatness to the threshold value determination unit 75.

  In the present embodiment, in order to solve the above-described problem of image quality degradation, processing for analyzing the flatness of an image near the target image and adaptively determining a threshold value according to the flatness as shown in FIG. Do. That is, when the flatness near the target pixel is low, the threshold value of the matching error is set large, and when the flatness is high, the threshold value of the matching error is set small. By doing in this way, it is possible to prevent the corresponding points of the pattern portion with low flatness from being used correctly and to use the incorrect corresponding points of the pattern portion with high flatness. As for the corresponding points, there are generally a plurality of corresponding points like pixel information indicated by triangles and diamonds in FIG. If there is no effective corresponding point, an interpolation image may be created from only the pixels of the own frame as in the prior art without using corresponding points. The interpolated image generation unit 44 does not necessarily generate an interpolated image after all the corresponding points are calculated. The interpolated image generating unit 44 adds these corresponding points in parallel with the calculation of new corresponding points. May be an embodiment in which an interpolation image is generated (changed).

  The flatness of the image can be obtained by taking the sum of 12 absolute values of adjacent differences in the vicinity of the target pixel as shown in FIG. Assuming that the sum of adjacent differences is S, the threshold value Th is determined by the following equation, for example. However, A is an arbitrary constant.

Therefore, when S is large, that is, when the flatness is low, the threshold value Th is large, and when S is small, that is, when the flatness is high, the threshold value Th is small.
The flatness of the image may be obtained by taking the sum of eight absolute values of differences between the target pixel and the peripheral pixels as shown in FIG. Further, the difference value between the pixel with the highest luminance in the vicinity of the target pixel and the pixel with the lowest luminance may be set as flatness. Furthermore, the frequency characteristic in the vicinity of the target pixel can be obtained by DFT (Discrete Fourier Transform), and the magnitude of the high frequency component can be made flat.

The threshold value Th may be determined by a function described more generally below, for example, in addition to the expression (3). FIG. 10 is an explanatory diagram showing a threshold setting method.
For example, if the inclination is A, the half line illustrated by the solid line in FIG. 10 corresponds to Expression (3). The other so-called piecewise linear function exemplified by the one-dot chain line is a monotonically increasing function in which the line segment for calculating Th is steeper than other sections of S while S is between the values of S1 and S2.

  FIG. 11 is a flowchart of the above processing in the corresponding point search unit 43 of the embodiment. First, the flatness analysis control units 77 and 78 calculate the flatness S respectively (step S10). Next, the flatness analysis control units 77 and 78 calculate threshold values corresponding to the calculated flatness S, respectively, and transmit the threshold values to the threshold value determination units 72 and 75 ( Step S20). On the other hand, each of the block matching units 71 and 74 calculates a matching error Me (step S30). Whether the calculated matching error Me is equal to or less than each of the transmitted threshold values is compared and determined by the threshold value determination units 72 and 75 (step S40). If the value is below the threshold, the corresponding point is used. That is, the decimal precision alignment units 73 and 76 perform alignment based on the corresponding points (step S50). If it is not less than the threshold value in step S40, the corresponding point is not used (step S60).

  As described above, in the resolution enhancement of moving images using multiple pieces of frame information, the threshold value is adaptively determined according to the flatness near the target pixel in the corresponding point search process using block matching, and the matching error is determined. The threshold values were compared to decide whether to adopt them as corresponding points.

  By adaptively determining the threshold value according to the flatness as described above, in this embodiment, the corresponding points of the pattern portion with low flatness are correctly used, and the erroneous corresponding points of the pattern portion with high flatness are used. It can be prevented from being used.

  In addition, this invention is not limited to the said embodiment, In the range which does not deviate from the summary, it can implement in various modifications. For example, the threshold value may be calculated by inputting the flatness from the flatness analysis unit by the threshold value determination unit.

  Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements according to different embodiments may be appropriately combined.

  41 ... Frame memory, 42 ... Frame memory, 43 ... Corresponding point search unit, 44 ... Interpolated image generation unit, 45 ... Panel, 71 ... Block matching unit, 72 ... Threshold determination unit, 73 ... Decimal precision alignment unit, 74: Block matching unit, 75: Threshold value determining unit, 76: Decimal precision positioning unit, 77 ... Flatness analysis control unit, 78 ... Flatness analysis control unit.

Claims (6)

A block matching unit that performs block matching between a plurality of frames and determines corresponding points of the blocks between the frames;
A threshold determination unit that compares a matching error output of the block matching unit with a threshold value and outputs a corresponding point of integer precision;
A decimal precision alignment unit that creates a decimal precision corresponding point when the integer precision corresponding point is input;
A flatness analysis control unit for calculating the flatness of the image of the block and controlling the threshold to be lower when the flatness is high than when the flatness is low;
An interpolation image generating unit that generates a higher-resolution interpolation image from the input frame signal based on the plurality of decimal precision corresponding points;
The interpolated image generation unit generates the interpolated image using the corresponding point of the decimal precision when the matching error output is determined to be equal to or less than the threshold value, and the matching error output is equal to or less than the threshold value. An image processing apparatus that generates the interpolated image from only pixels of its own frame without using the corresponding point of decimal precision when it is not determined.   The image processing apparatus according to claim 1, wherein the threshold value is calculated as a monotonically increasing function related to the low flatness.   The image processing apparatus according to claim 2, wherein the monotonically increasing function is a linear expression.   The image processing apparatus according to claim 3, wherein the linear expression is a proportional expression.   The image processing apparatus according to claim 2, wherein the monotonically increasing function is a piecewise linear function.   The image processing apparatus according to claim 1, further comprising a panel that displays the interpolation image generated by the interpolation image generation unit.

Images