JP2010108205A - Super resolution image creating method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce noise generation in super resolution image processing when a block image which is a unit image for motion detecting comparison does not have a uniform spacial illumination change but has a pitched illumination change. <P>SOLUTION: Each of two block images which are a source candidate and a destination candidate respectively and used for motion detecting comparison is divided into a plurality of small sub-block images, and an brightness average is calculated for each sub-block. An offset is calculated by subtracting a brightness average of a sub-block image of a candidate destination block image from a brightness average of a sub-block image of a candidate source block image, in the two sub-block images which are located in a same coordinate in a block image. Then, by adding each offset to brightness of each sub-block image of the candidate destination block image, a candidate destination block image is generated with an illumination change removed. The motion detecting comparison is performed by comparing the source block image with the candidate destination block whose illumination change is removed. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for creating an image having a higher resolution than the original resolution by performing super-resolution processing using two images.

  Super-resolution processing exists as a technique for creating an image having a resolution higher than the original resolution from two images. In this super-resolution processing, a motion search is performed by comparing pixel luminances for each block image in an image. Then, a sub-pixel level motion vector between pixels is obtained using a pixel level motion vector obtained by motion search.

  Using the sub-pixel level motion vector information, pixel reconstruction is performed on two images to create one super-resolution image. As a motion search method in the case where a change in illuminance occurs between two images due to movement of a shadow, a light source, or the like, for example, the luminance average value from the luminance of each pixel as disclosed in JP 2007-74592 A (Patent Document 1) There is a method for comparing data obtained by subtracting.

JP 2007-74592 A

The problems of the above prior art will be described below.
The spatial illuminance change in the block image, which is the unit for comparison, is not uniform. In this case, even if the average luminance value of the block image is subtracted from the luminance value of each pixel of the block image, there is a problem that the influence of the gradient cannot be removed and an erroneous motion search result is obtained.

  In addition, even if the conventional technology is applied to super-resolution, a super-resolution image is created from two images obtained by subtracting the luminance average value, so that a super-resolution image with the original luminance cannot be created and noise increases. There is a problem.

  An object of the present invention is to solve the above-described problems of the prior art and provide a super-resolution process in which block noise does not increase.

Of the inventions disclosed in the present application, the outline of typical ones will be briefly described as follows.
The two block images of the movement source and the movement destination candidate used for the motion search comparison are each divided into small sub-block images, and an average luminance value is calculated for each sub-block image. Of the two sub-block images located at the same coordinates in the block image, the average luminance value of the sub-block image of the destination candidate block image is subtracted from the average luminance value of the sub-block image of the source block image to obtain an offset. . Each offset is added to the luminance of each sub-block image of the movement destination candidate block image. As a result, a movement destination candidate block image in which the relative illuminance change between the two block images is removed is created. At the time of motion search comparison, comparison is performed using a source block image and a destination candidate block image from which the illuminance change is removed.

  Similarly, pixel reconstruction is performed using the movement source block image and the movement destination candidate block image from which the illuminance change is removed.

Among the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows.
By dividing the block image into sub-blocks and eliminating the difference in the average luminance value, even if there is a gradient in the illuminance in the block image that is the comparison unit, comparison and pixel reconstruction are performed without being affected by the gradient Therefore, an original super-resolution image that does not increase noise can be obtained.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

  With reference to FIG. 1, a system configuration to which an image super-resolution apparatus that creates an image having a higher resolution than the original resolution by using two images, which is a first embodiment of the present invention, will be described.

  In the present embodiment, super-resolution execution target image data 101 at time t = T used for creating a high-resolution image using super-resolution processing, and creating a high-resolution image using super-resolution processing. The super-resolution execution target image data 102 at the time t = T + 1 used for the motion search, the motion search region output unit 103 that outputs the motion search target region as a block image, and the motion destination candidate region in the motion search as a block image. A block image output unit 104 for outputting together with a motion vector, a sub-block image output unit 105 for dividing and outputting the block image into sub-block images, a luminance average value calculating unit 106 for calculating an average luminance of the image, and a luminance average value An offset calculation unit 107 that calculates the difference as an offset, and an offset output from the offset calculation unit 107 is a block image output unit 104. Offset addition unit 108 for adding to each pixel of the block image output from the image, comparison unit 109 for comparing the images and calculating the similarity, and a motion vector selection unit for selecting a pixel level motion vector from the similarity obtained from the comparison unit 109 110, a sub-pixel level alignment unit 111 for obtaining a sub-pixel level motion vector, and a pixel reconstruction unit 112 that performs pixel reconstruction to create a super-resolution image.

The data flow in the present embodiment will be described below.
The super-resolution execution target image data 101 at the time t = T at which resolution is increased by super-resolution processing is blocked in the motion search target area output 103, and the super-resolution execution target image data 102 at the time t = T when the illuminance difference occurs is blocked. Input to the image output unit 104.

  The motion search target region output unit 103 extracts a region to be subjected to super-resolution processing from the super-resolution execution target image data 101 at time t = T, and performs the super-resolution execution at time t = T of the block image. The coordinates in the target image data 101 are output as position information. The block image output unit 104 extracts a movement destination candidate area as a block image from the super-resolution execution target image data 102 at time t = T + 1.

  Also, using the position information input from the motion search target area output unit 103, the vertical and horizontal movement distances of the extracted block image are output as motion vectors. The sub-block image output unit 105 divides the block image input from the motion search target region output 103 and the block image output unit 104 into sub-block images of a smaller region and outputs them. The average brightness value calculation unit 106 calculates and outputs the average brightness value of each pixel of the input sub-block image.

  Of the sub-block images input from the two luminance average value calculation units 106, the offset calculation unit 107 calculates the luminance average of each sub-block image at time t = T + 1 from the luminance average value of each sub-block image at time t = T. Subtract the value and use it as the luminance offset of each sub-block image. The offset addition unit 108 adds the offset input from the offset calculation unit 107 to the sub-block of the block image input from the block image output unit 104.

  Thereby, a block image of t = T + 1 from which the illuminance difference between t = T and t = T + 1 is removed is created. The created t = T + 1 block image from which the illuminance difference is removed is output together with the motion vector. The comparison unit 109 compares the block image input from the motion search target region output unit 103 with the block image from which the illuminance difference input from the offset addition unit 108 is removed, and calculates the similarity. The calculated similarity is output together with the motion vector.

  The motion vector selection unit 110 selects one having a high similarity from the plurality of similarities input from the comparison unit 109, and outputs a motion vector paired therewith as a pixel level motion vector. This completes the pixel level alignment.

  The subpixel level alignment unit 111 outputs a motion vector that has been aligned at the subpixel level based on the motion vector input from the motion vector selection unit 110 as a subpixel level motion vector. The pixel reconstruction unit 112 improves the resolution of the block image output from the motion search target region output unit 103 by using the block image from which the illuminance difference output from the sub-pixel level motion vector and the offset addition unit 110 is removed. Perform pixel reconstruction.

  By applying this embodiment, it is possible to create a super-resolution image from which the influence of the illuminance gradient is removed.

  With reference to FIG. 2, a system configuration to which a super-image super-resolution apparatus that creates an image having a higher resolution than the original resolution using two images, which is a second embodiment of the present invention, will be described.

  This embodiment includes super-resolution execution target image data 201 at time t = T, super-resolution execution target image data 202 at time t = T + 1, a motion search region output unit 203 that outputs a motion search target region as a block image, In the motion search, a block image output unit 204 that outputs a movement destination candidate area as a block image together with a corresponding motion vector, an approximate block image creation unit 205 that approximates the luminance of each pixel in the block image to a linear function, and a luminance difference An offset calculating unit 206 that calculates the offset, an offset adding unit 207 that adds the offset output from the offset calculating unit 206 to each pixel of the block image output from the block image output unit 204, and compares the images to calculate the similarity. The pixel level motion vector from the similarity obtained from the comparison unit 208 and the comparison unit 208 Motion vector selection unit 209 for selecting the sub-pixel level positioning portion 210 for determining the sub-pixel level motion vector, and a pixel reconstruction unit 211 that performs pixel reconstruction to create a super-resolution image.

  The data flow in the present embodiment will be described below. The processing after the offset calculation unit 207 is the same as that in the first embodiment.

  Block the super-resolution execution target image data 202 at time t = T when the illuminance difference has occurred in the motion search target area output 203 with the super-resolution execution target image data 201 at time t = T to increase resolution by super-resolution processing. Input to the image output unit 204. The motion search target region output unit 203 extracts a region to be subjected to super-resolution processing from the super-resolution execution target image data 201 at time t = T, and performs the super-resolution execution at time t = T of the block image. The coordinates in the target image data 201 are output as position information.

  The block image output unit 204 extracts a movement destination candidate area as a block image from the super-resolution execution target image data 202 at time t = T + 1. Also, using the position information input from the motion search target region output unit 203, the vertical and horizontal movement distances of the extracted block images are output as motion vectors.

  The approximate block image creation unit 205 approximates the luminance of each pixel of the block image input from the motion search target region output unit 203 and the block image output unit 204 to a linear function using the least square method, and time t = T And the approximate block image at time t = T + 1.

  The offset calculation unit 206 calculates each pixel of the approximate block image at time t = T + 1 from the luminance of each pixel of the approximate block image at time t = T among the approximate block images input from the two approximate block image creation units 205. Is subtracted and output as an offset for each pixel. Thereafter, the same processing as in the first embodiment is performed.

  By applying this embodiment, errors due to discontinuous data at the boundary of the sub-block image of Embodiment 1 can be removed.

  With reference to FIGS. 3 and 4, a system configuration to which an image super-resolution device that creates an image having a higher resolution than the original resolution using two images, which is a third embodiment of the present invention, will be described. .

FIG. 3 is a block diagram of the entire embodiment.
In FIG. 3, super-resolution execution target image data 301 at time t = T, super-resolution execution target image data 302 at time t = T + 1, a motion search region output unit 303 that outputs a motion search target region as a block image, a block Sub-block image output unit 304 that divides and outputs an image into sub-block images, luminance average value calculation unit 305 that calculates an average luminance value of the image, block image output and luminance average value for each sub-block for reducing the amount of calculation The block image / luminance average value output unit 306 that calculates and outputs the difference, the offset calculation unit 307 that calculates the difference between the luminance average values as an offset, and the offset output from the offset calculation unit 307 as the block image / luminance average value output unit 306 Offset addition unit 308 that adds to each pixel of the block image output from, and compares the images to calculate the similarity A comparison unit 309, a motion vector selection unit 310 that selects a pixel level motion vector from the similarity obtained from the comparison unit 309, a sub-pixel level alignment unit 311 that calculates a sub-pixel level motion vector, and a super-resolution image The pixel reconstruction unit 312 performs pixel reconstruction.

FIG. 4 is an internal configuration diagram of the block image / brightness average value output unit 306.
In FIG. 4, a block image / luminance average value output unit 306 includes a sub-block image division position designation unit 401 that designates a sub-block division position of the input image, and a sub-block image output unit that divides the input image into sub-blocks and outputs the sub-block. 402, a luminance average value calculation unit 403 that calculates the average luminance value of the input image, a luminance average value storage unit 404 that stores the average luminance value calculated by the luminance average value calculation unit 403, and a sub-block image that is output as a block image A block area designating unit 405 for designating an area of the image, and an image extracting unit 406 for extracting an area in the range designated for the input image and outputting it as a block image.

The data flow in the present embodiment will be described below. The processing after the offset calculation unit 307 is the same as that in the first embodiment.
The super-resolution execution target image data 301 at time t = T at which resolution is increased by super-resolution processing is blocked in the motion search target area output 303, and the super-resolution execution target image data 302 at time t = T when the illuminance difference occurs is blocked. Input to the image output unit 306.

  The motion search target region output unit 303 extracts a region to be subjected to super-resolution processing from the super-resolution execution target image data 301 at time t = T, and performs the super-resolution execution at time t = T of the block image. The coordinates of the target image data 301 are output as position information.

  The block image / brightness average value output unit 306 extracts a movement destination candidate area from the super-resolution execution target image data 302 at time t = T + 1 as a block image. Also, using the position information input from the motion search target area output unit 303, the vertical and horizontal movement distances of the extracted block images are output as motion vectors. Moreover, the average value of the luminance of the block image to be output divided into sub-block images is output. Thereafter, the same processing as in the first embodiment is performed.

Next, the principle of the calculation amount reduction method performed by the block image / brightness average value output unit 306 will be described below with reference to FIGS.
The sub-block image division position specifying unit 401 sub-block division position information for dividing the entire super-resolution execution target image data 302 at time t = T + 1 input to the block image / luminance average value output unit 306 into sub-blocks. Is output.

  The sub-block image output unit 402 uses the sub-block division position information to divide the entire super-resolution execution target image data 302 at time t = T + 1 into sub-block images and output them (step 501). Also, the coordinates in the super-resolution execution target image data 302 at time t = T + 1 of the divided sub-block images are output as position information. The average brightness value calculation unit 403 calculates the average brightness value of the input sub-block and outputs it together with the position information (step 502).

  The luminance average value storage unit 404 stores the input luminance average value and position information as a pair. The sub-block image combination position specifying unit 405 outputs sub-block position information used for the combination in order to combine the plurality of sub-block images and output as a block image using the sub-block image division position information.

  The image extracting unit 406 uses the position information input from the sub-block image combination position specifying unit 405 and outputs a region including all the specified sub-block images as a block image. The average luminance value storage unit 404 outputs the average luminance value of the sub-block image corresponding to the position information input from the sub-block image designation unit 405.

  The sub-block image combination position specifying unit 405 outputs a plurality of block images by outputting position information of all combinations of sub-block images that can be configured as a block image (step 503). At this time, since the luminance average value only outputs the value stored in the luminance average value storage unit 404, the need for recalculation is eliminated and the calculation amount can be reduced.

  Sub-block image division position designation section 401 outputs new sub-block division position information obtained by shifting sub-block division positions after sub-block image combination position designation section 405 outputs position information of all combinations of sub-block images. (Step 504).

  Each module that has received new sub-block division position information repeats the same processing as the pixel level motion vector. The sub-block image division position designation unit 401 outputs all sub-block division position information that can be divided as sub-blocks.

  By applying this embodiment, it is possible to reduce the amount of calculation compared to the first embodiment by using a previously calculated average brightness value.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

For example, the following are possible:
In the sub-block image output unit 105 according to the first embodiment and the sub-block image output unit 402 according to the third embodiment, the width and height of the sub-block image are fixed, but it is possible to instruct these values from the outside. As a result, it becomes possible to cope with changes in illuminance and change the processing amount for each super-resolution target image.

  In the third embodiment, when the sub-block division position is changed, all of the average luminance values are recalculated. Among the pixels in the range of one sub-block image, the luminance of the pixels outside the sub-block image range is determined as the average luminance value. By subtracting from the above and adding the luminance of the pixels newly added to the sub-block image range to the luminance average value, the average value calculation processing for all the pixels is not necessary, and the calculation amount can be reduced as compared with the third embodiment. It is.

It is a block diagram of the 1st Example of this invention. It is a block diagram of the 2nd Example of this invention. It is a block diagram of the 3rd Example of this invention. It is a block diagram of one module in the 3rd Example of this invention. It is a figure explaining the principle of the calculation method reduction method of the 3rd Example of this invention.

Explanation of symbols

  101 ... Super-resolution execution target image data at time t = T, 102 ... Super-resolution execution target image data at time t = T + 1, 103 ... Motion search target region output unit, 104 ... Block image output unit, 105 ... Sub-block Image output unit 106 ... Average luminance value calculation unit 107 107 Offset calculation unit 108 Offset addition unit 109 109 Comparison unit 110 Motion vector selection unit 111 Sub-pixel unit alignment unit 112 Pixel reconfiguration 201: Super-resolution execution target image data at time t = T, 202 ... Super-resolution execution target image data at time t = T + 1, 203 ... Motion search target region output unit, 204 ... Block image output unit, 205 ... Approximate block image creation unit, 206 ... Offset calculation unit, 207 ... Offset addition unit, 208 ... Comparison unit, 209 ... Motion vector selection unit, 210 ... Subpixel unit alignment unit 211... Pixel reconstruction unit 301. Super-resolution execution target image data at time t = T 302. Super-resolution execution target image data at time t = T + 1 303. Motion search target region output , 304: Sub-block image output unit, 305: Luminance average value calculation unit, 306 ... Block image / luminance average value output unit, 307 ... Offset calculation unit, 308 ... Offset addition unit, 309 ... Comparison unit, 310 ... Motion vector Selection unit, 311 ... Subpixel unit alignment unit, 312 ... Pixel reconstruction unit, 401 ... Subblock image division position designation unit, 402 ... Subblock image output unit, 403 ... Luminance average value calculation unit, 404 ... Luminance average value Storage unit 405 ... Sub-block image combination designation unit 406 ... Image extraction unit 501 ... Sub-block division 502 ... Degrees average calculation, 503 ... block image output, 504 ... sub-block division position change

Claims (5)

In a super-resolution image creation method that uses two images to create an image with a resolution higher than the original resolution,
The source and destination candidate block images, which are comparison units for motion search, are divided into sub-block images,
Calculate the average brightness for each sub-block image,
The difference between the average values of the sub-block images corresponding to the same coordinate position in the block image is calculated as an offset,
Add the offset to the brightness of each sub-block image of the destination candidate block image to make the block image with illuminance change removed,
A method for creating a super-resolution image, characterized in that similarity comparison processing and pixel reconstruction are performed using a source block image and a block image from which illuminance change has been removed. In a super-resolution image creation method that uses two images to create an image with a resolution higher than the original resolution,
The luminance of the source and destination candidate block images, which are comparison units for motion search, is approximated to a linear function to obtain a source source approximate block image and a destination candidate approximate block image,
Calculate the luminance difference of each pixel of the source approximate block image and the destination candidate approximate block image as an offset,
Add the offset to the brightness of each pixel of the destination block image to make the block image with illuminance change removed,
A method for creating a super-resolution image, characterized in that similarity comparison processing and pixel reconstruction are performed using a source block image and a block image from which illuminance change has been removed. The super-resolution image creation method according to claim 1,
The entire image including the destination candidate image is divided into sub-block images in advance,
Save the brightness average value calculated for each sub-block image,
A super-resolution image generating method, comprising: outputting a block image composed of a combination of sub-block images and a stored brightness average value of the sub-block images. The super-resolution image creation method according to claim 3,
When recalculating the average luminance value of the sub-block image by changing the sub-block image division position, the luminance value of the pixel newly added to the sub-block image range is added to the average value,
A super-resolution image creation method, wherein a new luminance average value is calculated by subtracting a luminance value of a pixel outside the range of a sub-block image from the average value. In the super-resolution image creation method according to any one of claims 1, 3, and 5,
A super-resolution image creation method, wherein the width and height of a sub-block image are arbitrarily specified within a range not exceeding the width and height of a block image.

Images