SUPER RESOLUTION FROM MULTIPLE LOW RESOLUTION IMAGES Florin Manaila1 Costin-Anton Boiangiu2 Ion Bucur3 ABSTRACT Although the technology of optical instruments is constantly advancing, the capture of high resolution images is limited by both the shortcomings of the imaging devices and the law of physics (uncertainty principle applied onto photons or the wave-like theory of light). The current paper presents an algorithm for processing a set of images sharing the same subject with the purpose of extracting a higher resolution output image of the subject, using partial information from every one of the low resolutions samples in the input set. KEYWORDS: super resolution, SR from LR, imaging devices, resolution enhancement, image alignment. 1. INTRODUCTION Obtaining images of high resolution in domains like cartography or geology, but also in surveillance or national security, is a priority research domain. When optical devices hit a wall built by the technological limitations or by the law of physics, gathering a larger set of sample images or evidences (and their processing with the goal in mind to improve the final optical result) seems to be the only way towards progress. Obtaining the Super-Resolution (SR) [1][5] is possible both by using multiple sources or a single one. Building SR using multiple sources is based on a set of translations at sub-pixel level between multiple frames presenting the same subject. Using information from multiple low resolution images is created an output which presents a better visualization of the scene. On the other hand, obtaining Super-Resolution from a single image uses information from the same source in other areas or even from different images in order to decide how the final result should look like. Although, initially the Super-Resolution algorithms were used to enhance “black-and-white” pictures, advancements in this research field permitted their usage in the field of color imagery or even in three dimensional scenes. The current paper presents an algorithm for smart overlap of a set of Low-Resolution images in order to combine their partial sub-pixel information and to obtain a visually better result at twice the input resolution. Engineer, florin.manaila@cti.pub.ro, ”Politehnica” University of Bucharest, 060042 Bucharest, Romania Associate Professor PhD Eng., costin.boiangiu@cs.pub.ro, ”Politehnica” University of Bucharest, 060042 Bucharest, Romania 3 Associate Professor PhD Eng., ion.bucur@cs.pub.ro, ”Politehnica” University of Bucharest, 060042 Bucharest, Romania 1 2 2. TECHNICAL PRELIMINARIES ”Super Resolution” is the process of artificially increasing the resolution of an image using all means possible and all the available input data. Such SR algorithms in action were always shown in movies: a typical scene is when a PC operator gradually increases the size of an image and result details are coming, and coming, and coming… after every click. Figure 1. Purely fictional approach to Super-Resolution in Movies (Two frames from the movie The Bourne Identity showing the enhancement of a surveillance video) In such cases spots on the screen are transformed in clear people faces or valid car license numbers. Clearly, the previously presented situation is a purely fictional one, because, theoretically, there is an infinite number of images that fit into the low-resolution acquired data pixels. Starting from a single image cannot lead to a recovery of such a great amount of detail but using multiple images as a starting point, one can build a SR image with even twice the initial resolution on both axes. The SR technique is widely used in a variety of domains. NASA uses this technique to obtain detailed images of planets, stars and other points of interest. Other applications may reside in the area of identifying a license plate from a blurred or pixilated photograph or conversion from NTSC recordings to the new HD formats. There were proposed a variety of technical solutions for this problem: initially some frequency techniques were employed using the shift and aliasing properties of the Fourier transform. Afterwards, algorithms were developed which rely on a spatial approach of the problem. 3. BASIC SUPER-RESOLUTION ALGORITHM To emphasize the need for such a SR technique let’s imagine the following situation: we have at our disposition a camera with finite resolution of MxN and we need to obtain a 2Mx2N resolution. What solution can we imagine? One option is to acquire 4 different images with the existing camera, each image being offset with half a pixel in both X and Y directions. If we combine the images on a desired 2Mx2N resolution grid, we’ll end up with our near to “dream result” image. [4] 4. RELATED WORK The paper [6] describes the research carried in the field; there are many possible approaches for Super-Resolution; some of them will be briefly reminded in the following sections. Frequency Domain Approaches The SR problem has been treated with a frequency approach by Tsai and Huang in 1984. Before their contribution, the interpolation was the best technique available. Tsai and Huang wanted to enhance the resolution of the images acquired by satellites, where they have at their disposal a great number of translated input images. Spatial Domain Approaches There are a lot of approaches in the spatial domain, some of which are presented in the following:  Non-Uniform Interpolation: this is a naïve method which maps existing pixels on a large matrix and then interpolates the values to obtain a smaller image (but still SR).  Deterministic Regularization  Stochastic Reconstruction Methods Other Super Resolution Methods/Approaches  Projection Onto Convex Sets  Iterative Back Projection These methods usually come with different regularization techniques in order to compensate for the unknowns or errors that propagate through the system (ex: variations at sub-pixel level, like air turbulence [3], between low resolution images) [2] 5. IMAGE ALIGNMENT The image alignment will be obtained using three degrees of freedom for the SR samples provided at input: horizontal offset (an X axis displacement: DX), vertical offset (a Y axis displacement: DY) and the angle of rotation around origin. Since the angle of rotation is computed around the origin, a correction of horizontal and vertical displacements may be necessary for images considered rotated around their centers. The value of the rotation angle is specified in degrees (0-360 range) and the displacement values in pixels or pixel fractions. The information is sent to the SR engine by an alignment application and may be manually adjusted. Since the algorithm uses multiple images of the same subject it is important that these images to contain a sub-pixel displacement on at least one axis. The software application interface (Fig. 2) allows the fine tuning of the input parameters and as a result the rebuild of the target image. Figure 2. User Interface for specifying the input parameters 6. PROCESSING OF IMAGES In the image processing step a support image is generated, with twice the resolution on both axis. The support image pixels information is initialized to null values. An iterator is running on the support image pixels (further referred as “small pixels”), another one on the pixels from the LR images (further referred as “big pixels”) and the overlap area is computed (Fig. 3). Figure 3. Placing a low resolution image over a high resolution grid For the correct identification of the little pixel above the big pixels, the origin (Top-Left) of every little pixel should be considered and then compute the corresponding coordinates in the support image using the following formulas: x'  ( x cos  y sin  ) / 2  x y '  ( y cos  y sin  ) / 2  y Now taking into account the coordinates that corresponds in the LR image, the big “related” pixels are computed. We say that a big pixel is related with a small one if the big pixel contains relevant information from the small one. As a result 4 different cases may emerge (Fig. 4). Figure 4. Pixel position possibilities a) Coordinates of the little pixel are in area 1. All the surface of the little pixel is overlapping on the big pixel so this one is the only related pixel. b) Coordinates of the little pixel are in area 2. Surface of the small pixel is overlapping onto 2 big pixels so there will be 2 big pixels related to the small pixel. c) Coordinates of the little pixel are in area 3. Surface of the small pixel is overlapping onto 2 big pixels so there will be 2 big pixels related to the small pixel. d) Coordinates of the little pixel are in area 4. Surface of the small pixel is overlapping onto 4 big pixels so there will be 4 big pixels related to the small pixel. Once there is a mapping between related pixels, a percentage of the coverage for the surface of the small pixel with the overlap of a related big pixel shall be computed. Having this information the color of the small pixel is computed by performing a weighted average of the color of related big pixels and using as weights the coverage percentages. 7. OBTAINED RESULTS The advantages of this method rely in obtaining a reasonable result with only a few LR input images. Since the initial assumption was that there is needed only a double of the initial resolution enhancement (on both axis) the eventual increase of the number of input LR images may contribute little to the quality of the output. a. b. Figure 5. a. One of four input images b. Result 8. FUTURE WORK The main drawback of the presented algorithm is that it only takes e fraction of the color from the big pixel and puts it in the small pixel, but leaves the value of the big pixel intact. One way to overcome this shortcoming is to use a deblurring or deconvolution in order to compact the spread values of a big pixel on the surface of the small pixel.[7] Due to the nature of the involved equations these can be easily generalized to obtain a result with a multiplication factor greater than 2 on both axis. In this way the quality of the result should scale better with the number of input images. So we have two factors that can correlate to improve the output image quality: the number of the input LR images and the multiplication factor of the desired resolution. Increasing these 2 values in parallel may lead to a certain increase in quality of the output, as long as the necessary image alignment can be performed to the desired precision. Moreover, on a better quality image a final deconvolution step may further enhance the perceived quality. The computing of the related pixels as it was presented does not take into account the rotation angle. Adding this parameter to the equations may improve (only marginally, though) the quality of the output. For example in Fig. 4 (d) a rotation might reduce the number of related pixels from 4 to 2. 9. CONCLUSIONS The presented algorithm builds reasonable quality SR images even in a scenario of low input LR count. It can be improved in certain aspects offering a starting point for new research activities. In conclusion, the article underlined the idea that whenever the limitations of the current devices or the laws of physics lead to lower than necessary quality in the acquired information, a post-processing step of the acquired data may improve significantly the perceived quality of the desired output. ACKNOWLEDGEMENTS The authors would like to thank Alexandru Irimia for his support and assistance with this paper. 10. REFERENCES [1] Neil Alldrin, “Super-Resolution”, University of California, San Diego, Web Link: http:// vision.ucsd.edu/ ~nalldrin/ research/ superresolution/ superresolution.pdf, UCSD Research Exam Paper, 2006 [2] Farsiu, S.; Robinson, M.D.; Elad, M.; Milanfar, P., "Fast and robust multiframe super resolution," Image Processing, IEEE Transactions on , vol.13, no.10, pp.1327,1344, Oct. 2004, doi: 10.1109/TIP.2004.834669 [3] Shimizu, M.; Yoshimura, S.; Tanaka, M.; Okutomi, M., "Super-resolution from image sequence under influence of hot-air optical turbulence," Computer Vision and Pattern Recognition, 2008, CVPR 2008. IEEE Conference on , vol., no., pp.1,8, 23-28 June 2008 doi: 10.1109/CVPR.2008.4587525 [4] Russell C. Hardie, Kenneth J. Barnard, John G. Bognar, Ernest E. Armstrong, Edward A. Watson, “High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system”, Optical Engineering, Vol. 37, No. 1. (1998), pp. 247-260, doi:10.1117/1.601623 [5] M. Bertero and P. Boccacci. "Super-resolution in computational imaging". Micron, 34:265–273, October 2003. [6] S. Borman and R. Stevenson, Spatial Resolution Enhancement of Low-Resolution Image Sequences – A Comprehensive Review with Directions for Future Research", Technical report, University of Notre Dame, 1998 [7] Mihai Zaharescu, Costin-Anton Boiangiu, “Image Deblurring: Challenges and Solutions”, proceeding of the 12th International Conference on Circuits, Systems, Electronics, Control & Signal Processing (CSECS '13), At Budapest, Hungary