CN112132753B - Infrared image super-resolution method and system for multi-scale structure guide image - Google Patents

Abstract

The invention discloses an infrared image super-resolution method and system for a multi-scale structure-guided image. The invention registers an infrared image I _nir and a visible light image I _vis to obtain a registered infrared image I nir _-reg ; _vis obtains the visible light image I ⁱ _vis under a variety of down-sampling scales i by n downsampling, the image I _nir-reg is down-sampled to obtain the image I _nir-ds ; the image I ⁱ _vis is converted to the HSV color space, and the color is set Threshold T will carry out target classification; then take the image I _nir-ds as the initial current image to be filtered, carry out n iterations and use the visible light image I i _vis under a variety of downsampling scales as a guide to carry out joint bilateral filtering to obtain infrared image superimposed ^. resolution image. The invention can effectively improve the resolution of the infrared image through the visible light guide image, improve the visual effect, and has high practical application value.

Description

Infrared image super-resolution method and system for multi-scale structure-guided images

technical field

The invention relates to the technical field of image processing, in particular to an infrared image super-resolution method and system for multi-scale structure-guided images.

Background technique

Compared with traditional imaging methods, infrared thermal imaging has its unique advantages. It has strong anti-interference ability, is less affected by the environment, and is more sensitive to changes in thermal radiation, which can effectively obtain temperature information in the scene. Infrared thermal imaging technology mainly receives the thermal radiation energy from the target through the infrared detector and the optical imaging objective lens, and reflects it on the photosensitive element of the infrared detector, so that the camera can obtain the temperature information of different targets at the same time. With the maturity of imaging sensor technology, its cost is gradually reduced, so that its application scope has become more extensive, and it has good application value in transportation, construction, security and other fields. However, the density of most infrared detector arrays is still relatively low, resulting in low imaging resolution, and it is difficult to meet people's needs for high-resolution infrared images. If the high-resolution infrared image is obtained by improving the hardware, the cost of the camera will be greatly increased. Using algorithms to super-resolution the infrared image can effectively improve the resolution of the infrared image without increasing the hardware cost of the camera. high-resolution infrared images.

Currently, super-resolution techniques can be divided into two categories, one is learning-based methods and the other is reconstruction-based methods. The learning-based method requires a large number of high-resolution image construction learning libraries to learn the model, with the help of pre-training learning to find or establish the mapping relationship between low-resolution images and their corresponding high-resolution images, and extract high-frequency information, thereby Given a low-resolution image, the corresponding high-resolution image is obtained through an optimization method; while the reconstruction-based method estimates a high-resolution image from a single or multiple low-resolution images. Learning-based methods rely heavily on data, require high hardware, and have obvious limitations. Considering that the visible light image and infrared image of the same scene have a certain correspondence, using the high-resolution visible light image of the same scene to guide the image low-resolution infrared image super-resolution is conducive to the recovery of high-frequency information from the low-resolution infrared image, and realizes the The need for infrared image super-resolution.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention: in view of the above problems of the prior art, a method and system for infrared image super-resolution of multi-scale structure-guided images are provided. The visual effect has high practical application value.

In order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

An infrared image super-resolution method for multi-scale structure-guided images, comprising:

1) Register the infrared image I _nir with the visible light image I _vis of the same scene to obtain the registered infrared image I _nir-reg ;

2) Downsampling the visible light image I _vis n times to obtain the visible light image I ⁱ _vis under various downsampling scales i , and down-sampling the registered infrared image I _nir-reg to obtain the image I _nir-ds , the image The size of I _nir-ds is equal to the size of the visible light image I ^i-min _vis at the smallest downsampling scale i _min ;

3) Convert the visible light images I ⁱ _vis under various downsampling scales i to the HSV color space, set the color threshold T to classify the objects in the visible light images I ⁱ _vis under various downsampling scales i , and save them separately The index information d ⁱ _c in the visible light image I ⁱ _vis of different categories of objects c under various downsampling scales i and design the corresponding filter kernel; take the image I _nir-ds as the initial current image to be filtered, and initialize the sampling scale k is 1;

4) For the current image to be filtered, the designed filter kernel is selected according to the target type, and then the visible light images I ⁱ _vis under various downsampling scales i are used as guides for joint bilateral filtering respectively, and different guides are extracted according to the index information d ^ic respectively _. Image to the pixel under the corresponding coordinates of the image after the current image to be filtered is filtered, replace the pixel of the corresponding position on the original current image to be filtered before filtering, thereby generating the infrared super-resolution image I ⁱ _SR under the sampling scale k ;

5) Determine whether the sampling scale k is equal to the number of downsampling times n . If not, then upsample the infrared super-resolution image I ⁱ _SR under the sampling scale k to the size of the visible light image I ⁱ _vis under the sampling scale k+1 . As the new current image to be filtered, add 1 to the sampling scale k , and jump to step 4); otherwise, output the infrared super-resolution image I ⁱ _SR at the final sampling scale k as the result.

Optionally, registering the infrared image I _nir with the visible light image I _vis of the same scene in step 1) refers to multiplying the infrared image I _nir by the registration matrix H to obtain the registered infrared image I _nir-reg .

Optionally, step 1) includes the step of generating a registration matrix H before: dividing the infrared image I _nir and the visible light image I _vis of the same scene into image blocks of a specified size, and then performing feature detection on each image block, according to the infrared image. The registration matrix H is obtained by calculating the proportional relationship between the feature points of the image I _nir and the visible light image I _vis of the same scene.

Optionally, the downsampling of the visible light image I _vis by n times in step 2) means that downsampling is performed twice, and the image obtained by each downsampling is reduced to 1/2 of the original image.

Optionally, the filter kernel in step 3) is a Gaussian filter kernel, and the designed Gaussian filter kernel parameters include the size of the Gaussian filter kernel, the spatial standard deviation of the filter kernel for the target, and the Gaussian range standard deviation.

Optionally, when designing the corresponding filter kernel in step 3), the target targets include vegetation and non-vegetation, and the parameters of the Gaussian filter kernel designed for vegetation are: the filter kernel size is 3*3, and the spatial standard deviation of the vegetation filter kernel is 3. The standard deviation of the Gaussian range is 0.03; the parameters of the Gaussian filter kernel designed for non-vegetation are: the filter kernel size is 3*3, the spatial standard deviation of the non-vegetation filter kernel is 10, and the standard deviation of the Gaussian range is 0.03.

Optionally, the function expression for joint bilateral filtering in step 4) is as follows:

In the above formula, J _p represents the output at the target pixel position p , W _p represents the weight coefficient of the neighboring pixels, Ω represents the pixel group around the target pixel, and I _q represents the pixel position q around the target pixel position p in the current image to be filtered. , f , g are Gaussian weight distribution functions respectively, p is the target pixel position, q is a pixel position around the target pixel position p , I _guide-p is the pixel of the target pixel position p in the guide image, I _guide-q is the pixel at the pixel position q around the target pixel position p in the guide image.

In addition, the present invention provides an infrared image super-resolution system for multi-scale structure-guided images, comprising an interconnected microprocessor and a memory, the microprocessor being programmed or configured to perform the multi-scale structure-guided image analysis. Steps of an infrared image super-resolution method.

In addition, the present invention also provides an infrared image super-resolution system for multi-scale structure-guided images, comprising an interconnected microprocessor and a memory, the memory having stored therein is programmed or configured to execute the multi-scale structure-guided images A computer program for the infrared image super-resolution method.

In addition, the present invention also provides a computer-readable storage medium storing a computer program programmed or configured to perform the infrared image super-resolution method of the multi-scale structure-guided image.

Compared with the prior art, the present invention has the following advantages: the present invention can optimize the infrared image for the characteristics of low resolution, low contrast, blurred imaging and the like. Considering that the high-resolution visible light image has rich details and edge texture information, the present invention satisfies the requirement of implementing adaptive filtering for targets with different characteristics in the scene by adopting a multi-scale structure guidance method and introducing an adaptive filtering kernel. Guiding in a multi-scale way can effectively use the similarity between the guiding image and the original image, and gradually improve the resolution of the infrared image through a progressive method. Targeted processing reduces the loss of target edge texture and detail information, effectively improves the visual effect of infrared images, enhances image clarity, and significantly improves infrared image resolution.

Description of drawings

The accompanying drawings that form a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute improper limitations on the present application.

FIG. 1 is a basic flowchart of a method according to an embodiment of the present invention.

FIG. 2 is a flowchart of multi-scale guided filtering in an embodiment of the present invention.

Fig. 3 is the infrared image I _nir input in the embodiment of the present invention

Fig. 4 is the visible light image I _vis input in the embodiment of the present invention

FIG. 5 is an infrared super-resolution image output in an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described and described in detail below in conjunction with the flowcharts and the embodiments. Obviously, the described embodiments are the present invention. Some examples, but not all examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

As shown in FIG. 1 and FIG. 2 , the infrared image super-resolution method of the multi-scale structure-guided image in this embodiment includes:

1) Register the infrared image I _nir with the visible light image I _vis of the same scene to obtain the registered infrared image I _nir-reg ;

2) Downsampling the visible light image I _vis n times to obtain the visible light image I ⁱ _vis under various downsampling scales i , and down-sampling the registered infrared image I _nir-reg to obtain the image I _nir-ds , the image The size of I _nir-ds is equal to the size of the visible light image I ^i-min _vis at the smallest downsampling scale i _min ;

3) Convert the visible light images I ⁱ _vis under various downsampling scales i to the HSV color space, set the color threshold T to classify the objects in the visible light images I ⁱ _vis under various downsampling scales i , and save them separately The index information d ⁱ _c in the visible light image I ⁱ _vis of different categories of objects c under various downsampling scales i and design the corresponding filter kernel; take the image I _nir-ds as the initial current image to be filtered, and initialize the sampling scale k is 1;

4) For the current image to be filtered, the designed filter kernel is selected according to the target type, and then the visible light images I ⁱ _vis under various downsampling scales i are used as guides for joint bilateral filtering respectively, and different guides are extracted according to the index information d ^ic respectively _. Image to the pixel under the corresponding coordinates of the image after the current image to be filtered is filtered, replace the pixel of the corresponding position on the original current image to be filtered before filtering, thereby generating the infrared super-resolution image I ⁱ _SR under the sampling scale k ;

5) Determine whether the sampling scale k is equal to the number of downsampling times n . If not, then upsample the infrared super-resolution image I ⁱ _SR under the sampling scale k to the size of the visible light image I ⁱ _vis under the sampling scale k+1 . As the new current image to be filtered, add 1 to the sampling scale k , and jump to step 4); otherwise, output the infrared super-resolution image I ⁱ _SR at the final sampling scale k as the result.

In this embodiment, the infrared image I _nir and the visible light image I _vis of the same scene can be obtained at the same time by using the DJI Yu 2 dual-light version unmanned aerial vehicle, wherein the size of the infrared image I _nir is 640*480*3, and the visible light image of the same scene Image I _vis size is 4056*3040*3.

In this embodiment, the registration of the infrared image I _nir with the visible light image I _vis of the same scene in step 1) refers to the registration by block registration, that is, multiplying the infrared image I _nir by the registration matrix H , to obtain the registered infrared image Inir _-reg . In this embodiment, the relative positions of the two modal cameras on the UAV are fixed, so the infrared and visible images obtained at different heights can be registered only by obtaining the corresponding registration matrix H at different heights.

Considering the large difference in resolution of infrared and visible images, and the characteristics of blurred infrared image imaging, it is difficult to achieve registration directly by selecting feature points. Therefore, in this embodiment, step 1) includes the step of generating a registration matrix H before: The image I _nir and the visible light image I _vis of the same scene are respectively divided into image blocks of a specified size (in this embodiment, the image is divided into 4*4 image blocks of uniform size), and then feature detection is performed on each image block. According to the infrared image I The registration matrix H is obtained by calculating the proportional relationship between _nir and the feature points of the visible light image I _vis of the same scene.

Multiply the infrared image I _nir by the registration matrix H to get the registered infrared image I _nir-reg , the function expression is:

In the above formula, ( x ₁ , y ₁ , 1) ^T represents a pixel in the visible light image I _vis , and ( x ₂ , y ₂ , 1) ^T represents a pixel in the infrared image I _nir . According to the registration matrix H , the infrared image I _nir can be transformed into the visible light image I _vis to obtain the registered infrared image I _nir-reg .

In step 2) of this embodiment, the visible light image I _vis is down-sampled n times to obtain the visible light image I ⁱ _vis under various downsampling scales i , where i = 1, 2, 3,..., n +1 is represented as a visible light image The scale information of I _vis , n is the number of downsampling. In step 2) of this embodiment, the down-sampling of the visible light image I _vis by n times means that down-sampling is performed twice, and the image obtained by each down-sampling is reduced to 1/2 of the original image.

After converting the visible light images I i vis under various downsampling scales i to the HSV color space in step 3) of this embodiment, by setting the color threshold T, the visible light images I ⁱ _vis under various downsampling scales i are ^converted _into HSV color space. Target classification, and save the index information d ⁱ _c in the visible light image I ⁱ _vis of different categories of targets c under various downsampling scales i respectively, and design the corresponding filter kernel; according to the richness of the edge texture information of different kinds of targets, use different It reduces the problem of loss of edge texture information of some targets caused by using a single filter kernel for different targets.

In this embodiment, the filter kernel in step 3) is a Gaussian filter kernel, and the designed Gaussian filter kernel parameters include the size of the Gaussian filter kernel, the spatial standard deviation of the filter kernel for the target, and the Gaussian range standard deviation.

As an optional specific implementation manner, the pixel range corresponding to the HSV color space is

0.1176<H<0.3137, S>0.1569, V>0.1569

By setting the color threshold T, the image targets are divided into two categories: vegetation and non-vegetation. For vegetation, the spatial standard deviation during filtering is appropriately reduced, and the influence of distant pixels on central pixels is reduced, thereby reducing the loss of vegetation temperature information. Specifically, when designing the corresponding filter kernel in step 3) of this embodiment, the target targets include vegetation and non-vegetation, and the parameters of the Gaussian filter kernel designed for vegetation are: the filter kernel size is 3*3, and the vegetation filter kernel space standard The difference is 3, the standard deviation of the Gaussian range is 0.03; the Gaussian filter kernel parameters designed for non-vegetation are: the filter kernel size is 3*3, the spatial standard deviation of the non-vegetation filter kernel is 10, and the standard deviation of the Gaussian range is 0.03.

In this embodiment, the function expression for joint bilateral filtering in step 4) is as follows:

In the above formula, J _p represents the output at the target pixel position p , W _p represents the weight coefficient of the neighboring pixels, Ω represents the pixel group around the target pixel, and I _q represents the pixel position q around the target pixel position p in the current image to be filtered. f , g are Gaussian weight distribution functions respectively, p is the target pixel position, q is a pixel position around the target pixel position p , I _guide-p is the pixel of the target pixel position p in the guide image, I _guide-q is the pixel at the pixel position q around the target pixel position p in the guide image.

In this embodiment, the input infrared image I _nir is shown in Fig. 3 , and the input visible light image I _vis of the same scene is shown in Fig. 4 . Finally, n+ 1 times (specifically 3 times in this embodiment) are performed through iteration. Step 4 ), the final result (infrared super-resolution image) is shown in Figure 5.

To sum up, the infrared image super-resolution method of the multi-scale structure-guided image in this embodiment first obtains visible light and infrared images and performs registration; secondly, the visible light images at different scales are obtained by down-sampling the visible light images for many times. , and downsample the infrared image to the same size as the visible light minimum scale image; design an adaptive filter kernel to reduce the problem of loss of some target edge texture information caused by filtering with a single filter kernel; finally, guided by the visible light image at the same scale, combined with The adaptive filtering kernel performs multi-scale guided filtering on low-resolution infrared images. Before each filtering, the image to be filtered is upsampled to ensure that the resolution of the image to be filtered is consistent with the guided image. After multiple iterations of filtering, the Obtain infrared super-resolution images. The infrared image super-resolution method of the multi-scale structure guided image provided by the present invention utilizes the multi-scale structure information as a guide, and introduces an adaptive filtering kernel to reduce the loss of target edge texture and detail information, and effectively improve the visual effect of the infrared image. , which enhances image clarity and significantly improves infrared image resolution.

In addition, the present embodiment also provides an infrared image super-resolution system for multi-scale structure-guided images, comprising an interconnected microprocessor and a memory, the microprocessor being programmed or configured to perform the aforementioned multi-scale structure-guided image processing. Steps of an infrared image super-resolution method.

In addition, the present embodiment also provides an infrared image super-resolution system for multi-scale structure-guided images, including an interconnected microprocessor and a memory, wherein the memory is programmed or configured to execute the aforementioned multi-scale structure-guided images. A computer program for the infrared image super-resolution method.

In addition, the present embodiment also provides a computer-readable storage medium in which a computer program programmed or configured to execute the infrared image super-resolution method of the multi-scale structure-guided image is stored.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. The present application refers to flowcharts of methods, apparatus (systems), and computer program products according to embodiments of the present application and/or processor-executed instructions generated for implementing a process or processes and/or block diagrams in a flowchart. A means for the function specified in a block or blocks. These computer program instructions may also be stored in a computer readable memory capable of directing an imaging computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory result in an article of manufacture comprising instruction means, the The instruction means implement the functions specified in the flow or flows of the flowcharts and/or the block or blocks of the block diagrams. These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above are only the preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, and all technical solutions under the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims (10)

1. an infrared image super-resolution method of a multi-scale structure-guided image, is characterized in that, comprising: 1) Register the infrared image I _nir with the visible light image I _vis of the same scene to obtain the registered infrared image I _nir-reg ; 2) Downsampling the visible light image I _vis n times to obtain the visible light image I ⁱ _vis under various downsampling scales i , and down-sampling the registered infrared image I _nir-reg to obtain the image I _nir-ds , the image The size of I _nir-ds is equal to the size of the visible light image I ^i-min _vis at the smallest downsampling scale i _min ; 3) Convert the visible light images I ⁱ _vis under various downsampling scales i to the HSV color space, set the color threshold T to classify the objects in the visible light images I ⁱ _vis under various downsampling scales i , and save them separately The index information d ⁱ _c in the visible light image I ⁱ _vis of different categories of objects c under various downsampling scales i and design the corresponding filter kernel; take the image I _nir-ds as the initial current image to be filtered, and initialize the sampling scale k is 1; 4) For the current image to be filtered, the designed filter kernel is selected according to the target type, and then the visible light images I ⁱ _vis under various downsampling scales i are used as guides for joint bilateral filtering respectively, and different guides are extracted according to the index information d ^ic respectively _. Image to the pixel under the corresponding coordinates of the image after the current image to be filtered is filtered, replace the pixel of the corresponding position on the original current image to be filtered before filtering, thereby generating the infrared super-resolution image I ⁱ _SR under the sampling scale k ; 5) Determine whether the sampling scale k is equal to the number of downsampling times n . If not, then upsample the infrared super-resolution image I ⁱ _SR under the sampling scale k to the size of the visible light image I ⁱ _vis under the sampling scale k+1 . As the new current image to be filtered, add 1 to the sampling scale k , and jump to step 4); otherwise, output the infrared super-resolution image I ⁱ _SR at the final sampling scale k as the result. 2. The infrared image super-resolution method for multi-scale structure-guided images according to claim 1, wherein in step 1), registering the infrared image I _nir with the visible light image I _vis of the same scene refers to registering the infrared image. Inir is multiplied by the registration matrix H to obtain the _registered infrared image Inir _-reg . 3. The infrared image super-resolution method of multi-scale structure-guided images according to claim 2, characterized in that, before step 1), it comprises the step of generating a registration matrix H : combining the infrared image I _nir and the visible light image of the same scene I _vis is divided into image blocks of specified size respectively, and then feature detection is performed on each image block, and the registration matrix H is calculated according to the proportional relationship between the feature points of the infrared image I _nir and the visible light image I _vis of the same scene. 4. The infrared image super-resolution method for multi-scale structure-guided images according to claim 1, wherein in step 2), downsampling the visible light image I _vis by n times refers to performing 2 downsampling, and each time The image obtained by sub-sampling is reduced to 1/2 of the original image. 5. The infrared image super-resolution method for multi-scale structure guided images according to claim 1, wherein the filter kernel in step 3) is a Gaussian filter kernel, and the designed Gaussian filter kernel parameters include the Gaussian filter kernel's parameters. size, filter kernel spatial standard deviation and Gaussian range standard deviation for the target. 6. The infrared image super-resolution method for multi-scale structure-guided images according to claim 5, characterized in that, when designing the corresponding filter kernel in step 3), the target targets include vegetation and non-vegetation, and the vegetation is designed for The parameters of the Gaussian filter kernel are: the filter kernel size is 3*3, the spatial standard deviation of the vegetation filter kernel is 3, and the Gaussian range standard deviation is 0.03; the Gaussian filter kernel parameters designed for non-vegetation are: the filter kernel size is 3*3, The non-vegetation filter kernel has a spatial standard deviation of 10 and a Gaussian range standard deviation of 0.03. 7. The infrared image super-resolution method for multi-scale structure-guided images according to claim 5, wherein the functional expression for joint bilateral filtering in step 4) is shown in the following formula:

In the above formula, J _p represents the output at the target pixel position p , W _p represents the weight coefficient of the neighboring pixels, Ω represents the pixel group around the target pixel, and I _q represents the pixel position q around the target pixel position p in the current image to be filtered. f , g are Gaussian weight distribution functions respectively, p is the target pixel position, q is a pixel position around the target pixel position p , I _guide-p is the pixel of the target pixel position p in the guide image, I _guide-q is the pixel at the pixel position q around the target pixel position p in the guide image. 8. An infrared image super-resolution system for multi-scale structure-guided images, comprising an interconnected microprocessor and memory, wherein the microprocessor is programmed or configured to perform any one of claims 1 to 7 The steps of the infrared image super-resolution method of the multi-scale structure-guided image described in item. 9. An infrared image super-resolution system for multi-scale structure-guided images, comprising an interconnected microprocessor and a memory, wherein the memory is programmed or configured to perform any one of claims 1 to 7. A computer program for the infrared image super-resolution method of multi-scale structure-guided images. 10. A computer-readable storage medium, wherein the computer-readable storage medium stores therein an infrared image super-resolution that is programmed or configured to perform the multi-scale structure-guided image of any one of claims 1 to 7 A computer program for the rate method.

Images