CN114049248A - Image processing method, device, chip and storage medium for infrared image - Google Patents
Image processing method, device, chip and storage medium for infrared image Download PDFInfo
- Publication number
- CN114049248A CN114049248A CN202111355544.6A CN202111355544A CN114049248A CN 114049248 A CN114049248 A CN 114049248A CN 202111355544 A CN202111355544 A CN 202111355544A CN 114049248 A CN114049248 A CN 114049248A
- Authority
- CN
- China
- Prior art keywords
- image
- mode
- infrared
- images
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F15/00—Digital computers in general; Data processing equipment in general
- G06F15/76—Architectures of general purpose stored program computers
- G06F15/78—Architectures of general purpose stored program computers comprising a single central processing unit
- G06F15/7807—System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Microelectronics & Electronic Packaging (AREA)
- Image Processing (AREA)
Abstract
本发明公开了一种红外图像的图像处理方法、装置、系统级芯片及计算机可读存储介质，该方法应用于系统级芯片，包括：根据确定的目标图像模式获取探测器采集的待处理红外图像；根据系统级芯片中的硬件资源，确定目标图像模式对应的一个或多个硬件处理单元，以便将目标图像模式的图像处理流程拆分到各硬件处理单元中；利用一个或多个硬件处理单元，按照目标图像模式的图像处理流程对待处理红外图像进行图像处理，获取目标图像模式的输出图像；本发明采用探测器直驱的方式，利用Soc中不同的硬件资源完成复杂度较高且处理效果较好的红外成像，以在未采用FPGA的情况下，基于Soc实现了低成本、高质量和高效率的红外成像，提升了用户体验。
The invention discloses an image processing method, device, system-level chip and computer-readable storage medium for infrared images. The method is applied to the system-level chip and includes: acquiring an infrared image to be processed collected by a detector according to a determined target image mode ; According to the hardware resources in the system-level chip, determine one or more hardware processing units corresponding to the target image mode, so as to split the image processing flow of the target image mode into each hardware processing unit; Use one or more hardware processing units , perform image processing on the infrared image to be processed according to the image processing flow of the target image mode, and obtain the output image of the target image mode; the present invention adopts the method of direct drive of the detector, and uses different hardware resources in the Soc to complete the complex and the processing effect. Better infrared imaging, to achieve low-cost, high-quality and high-efficiency infrared imaging based on Soc without using FPGA, and improve user experience.
Description
Technical Field
The present invention relates to the field of infrared imaging computing, and in particular, to an image processing method and apparatus for infrared images, a system on chip, and a computer readable storage medium.
Background
The infrared imaging technology has the advantages of strong smoke penetration capability, capability of working all day long and the like, and has wide application prospect in the fields of military weapon allocation, industrial production, security monitoring and the like. However, with the rapid development of technology, the demands for low cost, high sensitivity systems and high definition infrared images are increasingly stringent for the application field.
The current infrared movement image equipment mostly completes an image processing process based on a programmable gate array (FPGA), has the defects of large volume, few extensible functions, high price and the like, and cannot meet the complex and changeable intelligent demand analysis in the industry by only using the movement equipment. Therefore, how to provide an infrared image processing method based on Soc (System on Chip, or System on Chip), how to utilize different hardware resources to complete an imaging scheme with higher complexity and better processing effect, and how to directly drive by using a detector to reduce the overall hardware cost is a problem which needs to be solved urgently nowadays.
Disclosure of Invention
The invention aims to provide an image processing method and device of an infrared image, a system-on-chip and a computer readable storage medium, so as to utilize different hardware resources in Soc to complete an imaging scheme with higher complexity and better processing effect, and the overall hardware cost can be reduced by adopting a direct drive mode of a detector.
In order to solve the above technical problem, the present invention provides an image processing method for infrared images, which is applied to a system-on-chip, and comprises:
determining a current target image mode, and acquiring an infrared image to be processed, which is acquired by a detector, according to the target image mode;
determining one or more hardware processing units corresponding to the target image mode according to hardware resources in the system-on-chip so as to divide an image processing flow of the target image mode into the hardware processing units;
performing image processing on the infrared image to be processed according to the image processing flow of the target image mode by using the one or more hardware processing units to obtain an output image of the target image mode; wherein the image processing flow comprises at least one of image enhancement, non-uniformity correction and image noise reduction.
Optionally, when the image processing flow includes the image enhancement, the non-uniformity correction, and the image noise reduction, and the plurality of hardware processing units corresponding to the target image mode include an ARM, a DSP, and an ISP, the performing, by using the one or more hardware processing units, image processing on the infrared image to be processed according to the image processing flow of the target image mode to obtain an output image of the target image mode includes:
performing the non-uniformity correction on the infrared image to be processed by utilizing the ARM to obtain a corrected image;
performing image 2D noise reduction on the corrected image by using the ISP to obtain a noise-reduced image;
utilizing the DSP to perform image enhancement corresponding to the target image mode on the noise-reduced image to obtain an enhanced image;
and carrying out image 3D noise reduction on the enhanced image by utilizing the ISP to obtain the output image.
Optionally, when the target image mode is a wide dynamic mode, the performing, by using the DSP, image enhancement corresponding to the target image mode on the noise-reduced image to obtain an enhanced image includes:
carrying out high-pass filtering on a preset number of noise-reduced images to obtain a preset number of high-frequency images; the noise reduction image is an image corresponding to the infrared image to be processed of different integration times of the preset number of frames acquired by the detector;
performing Gaussian filtering on the high-frequency images to obtain the preset number of weight images;
performing low-pass filtering on the noise-reduced images to obtain the preset number of base layer images;
acquiring the preset number of detail images according to the noise reduction image and the base layer image; wherein the detail image is the difference between each noise-reduced image and the corresponding base layer image;
and fusing the base layer image and the detail image according to the weight image to obtain a frame of the enhanced image.
Optionally, when the preset number is 3, the fusing the base layer image and the detail image according to the weight image to obtain a frame of the enhanced image includes:
calculating the enhanced image by fusion (w1 f1+ w2 f2+ w3 f 3)/w; wherein fusion is the enhanced image, f1 ═ base1+ detail1, f2 ═ base2+ detail2, f3 ═ base3+ detail3, w ═ w1+ w2+ w3, w1, w2, and w3 are the weighted images, base1, base2, and base3 are the base layer images, and detail1, detail2, and detail3 are the detail images.
Optionally, when the non-uniformity correction includes a fringe non-uniformity correction, the performing the non-uniformity correction on the infrared image to be processed by using the ARM to obtain a corrected image includes:
utilizing the ARM to carry out the stripe non-uniformity correction on the infrared image to be processed to obtain a corrected image; and the infrared image to be processed is an image which is output by the detector and is subjected to response consistency correction.
Optionally, when the target image mode is a wide dynamic mode, before acquiring the to-be-processed infrared image collected by the detector according to the target image mode, the method further includes:
sending the image frame output parameters corresponding to the wide dynamic mode to the detector; wherein the image out-of-frame parameter comprises the preset number of different integration times.
Optionally, before the performing, by using the one or more hardware processing units, image processing on the infrared image to be processed according to the image processing flow of the target image mode and acquiring the output image of the target image mode, the method further includes:
acquiring image configuration parameters corresponding to the target image mode;
sending the image configuration parameters to respective corresponding hardware processing units;
correspondingly, the performing, by using the one or more hardware processing units, image processing on the infrared image to be processed according to the image processing flow of the target image mode to obtain an output image of the target image mode includes:
and performing image processing of the image processing flow on the infrared image to be processed according to the image configuration parameters by using the one or more hardware processing units to obtain an output image of the target image mode.
The invention also provides an image processing device of the infrared image, which is applied to a system-on-chip and comprises:
the image acquisition module is used for determining a current target image mode and acquiring an infrared image to be processed, which is acquired by the detector, according to the target image mode;
a hardware determining module, configured to determine one or more hardware processing units corresponding to the target image mode according to hardware resources in the system-on-chip, so as to split an image processing flow of the target image mode into the hardware processing units;
the image processing module is used for carrying out image processing on the infrared image to be processed according to the image processing flow of the target image mode by utilizing the one or more hardware processing units to obtain an output image of the target image mode; wherein the image processing flow comprises at least one of image enhancement, non-uniformity correction and image noise reduction.
The present invention also provides a system-in-a-chip comprising:
a memory for storing a computer program;
a processor for implementing the steps of the image processing method for infrared images as described above when executing the computer program.
Furthermore, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, realizes the steps of the image processing method for infrared images as described above.
The invention provides an image processing method of an infrared image, which is applied to a system-on-chip and comprises the following steps: determining a current target image mode, and acquiring an infrared image to be processed, which is acquired by a detector, according to the target image mode; determining one or more hardware processing units corresponding to the target image mode according to hardware resources in the system-on-chip so as to divide the image processing flow of the target image mode into the hardware processing units; performing image processing on the infrared image to be processed according to the image processing flow of the target image mode by using one or more hardware processing units to obtain an output image of the target image mode; wherein the image processing flow comprises at least one of image enhancement, non-uniformity correction and image noise reduction;
therefore, the infrared imaging method and the infrared imaging device have the advantages that the one or more hardware processing units corresponding to the target image mode are utilized, the image processing is carried out on the infrared image to be processed according to the image processing flow of the target image mode, the output image of the target image mode is obtained, the direct driving mode of the detector can be adopted, the imaging with higher complexity and better processing effect can be completed by utilizing different hardware resources in the Soc, the infrared imaging with low cost, high quality and high efficiency is realized on the basis of the Soc under the condition that the FPGA is not adopted, and the user experience is improved. In addition, the invention also provides an image processing device of the infrared image, a system-on-chip and a computer readable storage medium, and the beneficial effects are also achieved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of an image processing method for infrared images according to an embodiment of the present invention;
FIG. 2 is a schematic view illustrating an image processing flow in a heaven and earth mode of another infrared image processing method according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating an output image of a world mode of another infrared image processing method according to an embodiment of the present invention;
fig. 4 is a schematic diagram illustrating an image processing flow in a wide dynamic mode of another infrared image processing method according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating an output image of a wide dynamic mode according to another exemplary embodiment of the present invention;
fig. 6 is a block diagram of an image processing apparatus for infrared images according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart illustrating an image processing method for infrared images according to an embodiment of the present invention. The method is applied to a system-on-chip, and comprises the following steps:
step 101: and determining a current target image mode, and acquiring the infrared image to be processed acquired by the detector according to the target image mode.
Specifically, the target image mode in this embodiment may be a preset image mode currently used, and if the number of the preset image modes is 1, the target image mode may be the preset image mode; when the number of the preset image modes is greater than 1, the target image mode may be any preset image mode. The preset image mode in this embodiment may be a preset image mode of infrared imaging, that is, an image processing mode of different imaging styles adopted by Soc.
It should be noted that, for the specific number and type of the preset image patterns in the present embodiment, the specific number and type may be set by a designer according to a practical scene and a user requirement, for example, the number of the preset image patterns may be 1, and the preset image pattern may be a sky-ground pattern, a wide dynamic pattern, or a general pattern. The number of the preset image modes may also be greater than 1, for example, the preset image modes may include a sky-ground mode, a wide dynamic mode and a normal mode, and the preset image modes may also include an outdoor mode, so that a user may select any one of the preset image modes (i.e., a target image mode) for image processing according to a specific application scene and own requirements; if the scenes such as frontier defense, sea defense or outdoor security protection are mostly half a day and half a ground, the heaven and earth mode can be selected; in a scene where industrial type applications require that a substantial portion of the objects within the field of view be seen, a wide dynamic mode may be selected.
Correspondingly, for the specific mode of determining the current target image mode by the Soc in this step, the specific mode can be set by a designer according to a practical scene and user requirements, and if the number of the preset image modes is 1, the Soc can directly determine the preset image mode as the target image mode. When the number of the preset image modes is greater than 1, the Soc may determine a target image mode from the preset image modes according to the acquired image mode selection instruction, that is, determine the preset image mode corresponding to the image mode selection instruction as the target image mode; that is, the user can select the required preset image mode by triggering the image mode selection instruction; the Soc may also perform scene recognition on the to-be-processed infrared image acquired by the detector, determine a preset image mode corresponding to the recognized scene as a target image mode, and determine the world mode as the target image mode if the recognized scene is a half-day-and-half-earth scene. The present embodiment does not set any limit to this.
Wherein, the detector in this step can be an infrared module for collecting infrared images, such as an uncooled infrared detector; the infrared image to be processed in this step may be an infrared image collected by the detector and output to the Soc (system on chip), that is, an original infrared image for infrared imaging needs to be processed by the Soc.
Correspondingly, the embodiment does not limit the specific type of the infrared image to be processed, for example, when the detector is an uncooled infrared detector (e.g., an uncooled infrared focal plane detector), the infrared image to be processed may be specifically an infrared image acquired by the uncooled infrared detector.
Similarly, the specific manner in which the Soc acquires the to-be-processed infrared image acquired by the detector according to the target image mode in this step may be set by a designer according to a practical scene and a user requirement, for example, when the number of the preset image modes is 1 in this embodiment, the Soc may directly acquire the to-be-processed infrared image acquired by the detector, that is, the detector may directly output the acquired infrared image (that is, the to-be-processed infrared image) to the Soc according to the preset image mode corresponding image framing parameter. When the number of the preset image modes is more than 1, the Soc can firstly send image frame output parameters corresponding to the target image mode to the detector to complete the configuration of the detector, and then receive the infrared image to be processed, which is acquired by the detector according to the received image frame output parameters; the detector outputs the collected infrared image (namely the infrared image to be processed) to the Soc according to the image framing parameters corresponding to the target image receiving mode; for example, when the target image mode is the wide dynamic mode, the Soc may obtain an image out-frame parameter corresponding to the wide dynamic mode (i.e., the target image mode) before this step; sending an image frame output parameter corresponding to the wide dynamic mode to a detector; the image out-frame parameters corresponding to the wide dynamic mode may include different integration times of a preset number (i.e., out-frame number), so that the Soc may perform image processing on the infrared images of different integration times of a preset number of frames (e.g., 3 frames) acquired by the detector in the wide dynamic mode to obtain 1 frame of output images.
Correspondingly, the preset number can be greater than or equal to 2, so that when the target image mode is the wide dynamic mode, the infrared images to be processed in different states are obtained by adopting multi-frame input of a preset number of frames and through the integration time configuration of the detector. As for the same target, the integration time is continuously increased, the output target is continuously enhanced, the target is gradually clear, and when the integration time is too long, the supersaturation phenomenon can occur; in comprehensive consideration, the preset number in this embodiment may be specifically 3, so as to obtain three frame inputs by using three different integration times of the detector; the short integration time of the three different integration times can ensure that the details of the object in the high-temperature area are displayed, the middle integration time can ensure that the whole picture is normally displayed, and the long integration time can ensure that the details of the object in the low-temperature area are displayed; different detectors are configured with different integration times according to respective noise-equivalent temperature differences (NETD), which is not limited in this embodiment.
Specifically, the Soc in this embodiment may include a plurality of functional modules, such as a detector configuration module, a data stitching module, and a shutter correction module, so as to implement detector driving and obtain the to-be-processed infrared image acquired by the detector.
Step 102: and determining one or more hardware processing units corresponding to the target image mode according to hardware resources in the system-on-chip so as to split the image processing flow of the target image mode into the hardware processing units.
It can be understood that, in this step, the one or more hardware processing units corresponding to the target image mode may be hardware processing resources in the Soc that complete image processing corresponding to the target image mode; in this embodiment, the image processing flow corresponding to the target image mode is split into the hardware processing units, so that the hardware processing units are used to complete the whole image processing flow of the target image mode, thereby implementing infrared imaging based on Soc.
Specifically, the specific number and type of the hardware Processing units corresponding to the target Image mode in this step may be set by a designer according to a practical scene and a user requirement, for example, the hardware Processing units corresponding to the target Image mode may include at least one of an ARM (Advanced RISC machine), a DSP (Digital Signal Processing, Digital Signal processor), a GPU (Graphics Processing Unit), an ISP (Image Signal Processing, Image Signal Processing Unit), and an NPU (Neural network processor). For example, if only ARM resources are available in the Soc, the entire image processing flow may run in the ARM, and if the hardware processing unit corresponding to the target image mode only includes the ARM, the ARM may adopt a multithreading combination NEON (ARM architecture processor extension structure) structure to perform image processing on the infrared image to be processed according to the image processing flow, and acquire an output image of the target image mode; if ARM + DSP resources are available in the Soc, the whole image processing flow can be split into the ARM and the DSP; if ARM + DSP + NPU resources are available in the Soc, the whole image processing flow can be split into the ARM, the DSP and the NPU, a neural network model can be used for replacing partial image processing flow, and the whole image processing flow can be completed more efficiently. The present embodiment does not set any limit to this.
It should be noted that, for the specific manner of determining one or more hardware processing units corresponding to the target image mode according to the hardware resources in the system-on-chip in this step, the specific manner may be set by a designer according to a practical scene and a user requirement, and when the hardware processing unit corresponding to each preset image mode is preset, the Soc may determine the hardware processing unit corresponding to the preset image mode determined as the target image mode as the hardware processing unit corresponding to the target image mode; the Soc may also detect its own hardware resources, and determine target resources (such as ARM, DSP, NPU, ISP, and the like) existing in the hardware resources as hardware processing units corresponding to the target image mode.
Step 103: performing image processing on the infrared image to be processed according to the image processing flow of the target image mode by using one or more hardware processing units to obtain an output image of the target image mode; wherein the image processing flow comprises at least one of image enhancement, non-uniformity correction and image noise reduction.
It can be understood that, in this step, the Soc may perform image processing on the infrared image to be processed according to the image processing flow corresponding to the target image mode by using the hardware processing unit corresponding to the target image mode, obtain an output image of the target image mode, and implement infrared imaging based on the Soc.
Correspondingly, the image processing flow in this step may be a processing flow of infrared imaging completed by the hardware processing unit. The specific process configuration of the image processing flow in the step can be set by a designer according to a practical scene and user requirements, for example, the image processing flow can comprise image enhancement so as to ensure the infrared imaging of the image style of a target image mode; in order to improve the imaging effect of the infrared imaging, the image processing flow may further include non-uniformity correction and/or image noise reduction (e.g., image 2D noise reduction and/or image 2D noise reduction). The present embodiment does not set any limit to this.
Specifically, the specific mode of performing image processing on the infrared image to be processed by the Soc according to the image processing flow by using one or more hardware processing units corresponding to the target image mode in the step to obtain the output image of the target image mode can be set by a designer, and if the image processing flow includes image enhancement, non-uniformity correction and image noise reduction, the Soc in the step can perform non-uniformity correction on the infrared image to be processed to obtain a corrected image; performing image 2D noise reduction on the corrected image to obtain a noise-reduced image; performing image enhancement corresponding to a target image mode on the noise-reduced image to obtain an enhanced image; performing image 3D noise reduction on the enhanced image to obtain an output image; wherein the image denoising comprises image 2D denoising and image 3D denoising; that is, the Soc may complete an image processing flow including non-uniformity correction, image 2D noise reduction, image enhancement, and image 3D noise reduction by using a hardware processing unit, and obtain an output image in the target image mode. For example, Soc may use ARM to perform non-uniformity correction on the infrared image to be processed, so as to obtain a corrected image; performing image 2D noise reduction on the corrected image by using an ISP (internet service provider) to obtain a noise-reduced image; utilizing the DSP to perform image enhancement corresponding to the target image mode on the noise-reduced image to obtain an enhanced image; and performing image 3D noise reduction on the enhanced image by using the ISP to obtain an output image.
It should be noted that the process of performing non-uniformity correction on the infrared image to be processed to obtain the corrected image may be set by a designer, for example, the non-uniformity correction may include response consistency correction and fringe non-uniformity correction, that is, Soc may perform response consistency correction on the infrared image to be processed to obtain a consistency corrected image; and performing stripe non-uniformity correction on the consistency correction image to obtain a correction image. When the detector performs response consistency correction on the output infrared image to be processed, the non-uniformity correction in the embodiment may also only include fringe non-uniformity correction, that is, the Soc may perform fringe non-uniformity correction on the infrared image to be processed to obtain a corrected image; the infrared image to be processed is an image which is output by the detector and is subjected to response consistency correction; for example, the Soc can utilize the ARM to perform non-uniformity correction on the infrared image to be processed, so as to obtain a corrected image.
Specifically, the above-mentioned process of response consistency correction can be implemented by using a calibrated response consistency correction method in the prior art, which is not limited in this embodiment. Since the non-refrigeration type infrared focal plane detector usually has the purpose of saving the production cost, each pixel does not have an independent amplifier, but the detection units in the same column or row share the same reading circuit; analyzing by taking an uncooled focal plane array in a column integration mode as an example, the integrated currents of the detection units in the same column correspond to the same bias voltage, and due to the existence of bias voltage noise of the amplifier, when the two columns are integrated, the voltages obtained by the grid electrodes are different, so that even if the same radiation value is obtained, the integrated currents flowing through each detection unit in the same column are different; meanwhile, the amplification factors of two amplifiers in different columns are not completely the same but have smaller difference, so that the output values have smaller difference; therefore, the correction of the fringes in the image is completed by the above-described process of the fringe non-uniformity correction with respect to the fringes existing in the uncooled focal plane array in this embodiment.
Correspondingly, the stripe non-uniformity correction process in this embodiment may beThe method comprises the following steps: calculating horizontal gradient map of image to be subjected to fringe non-uniformity correction (such as infrared image to be processed) by using hardware processing unit (ARM or DSP) corresponding to fringe non-uniformity correction, and counting gradient difference histogram, wherein the absolute value of difference can be limited to 0-255, for example, using histogram [ n ]]＝MIN(abs(src[i+1]-src[i]) 255) obtaining a gradient difference histogram through statistics, wherein n can be 0-255 statistical data, src is an input image (such as a horizontal gradient map), and i is an image traversal coordinate point; obtaining the gradient threshold by using the gradient threshold determination parameter in the issued image configuration parameters, for example, when the gradient threshold determination parameter is 0.85, the method can adopt
Acquiring a gradient threshold, wherein width and height are width and height of image resolution respectively; if the accumulated value is larger than the threshold (0.85 width height) corresponding to the gradient threshold determination parameter, outputting n as the gradient threshold; determining parameters according to the stripe intensity in the issued image configuration parameters, and acquiring an image stripe intensity value; and performing stripe non-uniformity correction on the to-be-stripe non-uniformity corrected image according to the image stripe intensity value and the gradient threshold value to obtain a corrected image.
When the image processing flow in this embodiment includes image denoising, the image denoising may include image 2D denoising and image 3D denoising, and the process of image 2D denoising in this embodiment may include: utilizing a hardware processing unit (such as an ARM, a DSP or an ISP) corresponding to the image 2D noise reduction, performing corresponding processing according to different noise types to complete spatial domain noise reduction (namely image 2D noise reduction), for example, performing Poisson noise reduction on the image to be subjected to 2D noise reduction (such as the corrected image) to obtain a Poisson noise reduction image so as to remove Poisson noise caused by photon signals in the image to be subjected to 2D noise reduction; performing burr noise reduction on the Poisson noise reduction image to obtain a burr noise reduction image so as to remove burr noise with abnormal and abrupt edges; and performing Gaussian noise reduction on the burr noise reduction image to obtain a noise reduction image so as to remove Gaussian noise. The 3D image denoising process in this embodiment may be implemented by using a hardware processing unit (such as an ARM, a DSP, or an ISP) corresponding to the 3D image denoising, in a manner the same as or similar to the 3D image denoising (i.e., a temporal domain denoising) method in the prior art, which is not limited in this embodiment.
Specifically, in this embodiment, each preset image mode has a great difference in the image enhancement process, and the hardware processing unit (such as an ARM, a DSP, or a GPU) corresponding to each preset image mode may perform the respective corresponding image enhancement process according to different image enhancement parameters in the delivered image configuration parameters. For example, when the target image mode is the normal mode, the hardware processing unit (such as an ARM, a DSP, or an ISP) corresponding to the normal mode may first divide the image to be enhanced (such as the above noise-reduced image) into m × m blocks to obtain a histogram, where the size of each block is width/m and height/m; then, counting the maximum pixel value maxValue and the minimum pixel value minValue of each block; then, alignment and linear mapping are carried out, and a mapping result dst is obtained as (src-minValue)/(maxValue-minValue), wherein src is an input image (such as a histogram); performing q-q mean filtering on the mapped mapping table to obtain a filtered image so as to prevent the block from generating an over-heavy halo phenomenon, for example, a mapping value corresponding to each pixel value of a first block is determined by taking the mean value of the mapping tables of q blocks in the neighborhood; performing threshold segmentation on the filtered image to obtain a foreground region and a background region so as to reserve the foreground region of a highlight region and only enhance a significant region concerned by human eyes in the image; and carrying out equalization processing on a foreground area in a histogram corresponding to the image to be enhanced to obtain an enhanced image, carrying out gray level resetting according to a mapping table of the foreground, carrying out no special processing on the background, achieving the effects of dark background and bright foreground, and finishing the block contrast enhancement of the image to be enhanced.
When the target image mode is the sky-ground mode, histogram statistics can be performed on an image to be enhanced (such as the noise-reduced image) by using a hardware processing unit (such as an ARM, or an ARM and a DSP, or an ARM and a GPU) corresponding to the sky-ground mode, an image vertical gradient difference histogram is calculated, and a cutting threshold value for cutting a sky area and a ground area is determined by combining the histogram corresponding to the image to be enhanced obtained through statistics and the image vertical gradient difference histogram obtained through calculation; cutting the image to be enhanced by utilizing a cutting threshold value, and determining a sky area and a ground area in the image to be enhanced; by using a preset fixed gray scale distribution range (such as 10-20 gray scales), the gray scales outside the preset fixed gray scale distribution range in the sky region in the image to be enhanced are redistributed, and the enhanced image is obtained after the fusion with the ground region, so that the output image of the sky-ground mode shown in fig. 3 is obtained.
When the target image mode is a wide dynamic mode, a hardware processing unit (such as an ARM, a DSP or a GPU) corresponding to the wide dynamic mode is utilized to perform high-pass filtering on a preset number of images to be enhanced (such as the noise reduction images) to obtain a preset number of high-frequency images; performing Gaussian filtering on the high-frequency images to obtain a preset number of weight images; carrying out low-pass filtering on the noise-reduced image to obtain a preset number of base layer images; acquiring a preset number of detail images according to the noise-reduced images and the base layer images, wherein the detail images are the difference between each noise-reduced image and the corresponding base layer image; according to the weighted image, fusing the base layer image and the detail image to obtain a frame of enhanced image so as to fuse the images with different integration time, thereby obtaining an output image with a wide dynamic mode as shown in fig. 5; the to-be-enhanced images are images corresponding to the to-be-processed infrared images of different integration times of a preset number of frames acquired by the detector. If the preset number is 3, that is, when the detector collects and outputs 3 frames of infrared images to be processed with different integration times to the Soc, the above-mentioned fusing the base layer image and the detail image according to the weight image to obtain a frame of enhanced image may include: calculating to obtain an enhanced image through fusion (w1 f1+ w2 f2+ w3 f 3)/w; wherein fusion is an enhanced image, namely an output result of image enhancement in a wide dynamic mode; f1 base1+ detail1, f2 base2+ detail2, f3 base3+ detail3, w1+ w2+ w3, w1, w2, and w3 are weight images, base1, base2, and base3 are base layer images, and detail1, detail2, and detail 3.
Further, as shown in fig. 2, when the target image mode is the world mode, the Soc may perform world scene analysis on the image to be enhanced before performing image enhancement on the image to be enhanced by using the hardware processing unit corresponding to the world mode, detect whether the image to be enhanced is in a world scene, and perform image enhancement on the image to be enhanced by using the hardware processing unit corresponding to the world mode if the image to be enhanced is in the world scene; if not, the hardware processing unit corresponding to the common mode is utilized to carry out image enhancement on the image to be enhanced.
Specifically, in this embodiment, the Soc may obtain an image configuration parameter corresponding to the target image mode before step 103; and sending the image configuration parameters to the respective corresponding hardware processing units, so that the hardware processing units can be utilized to perform image processing of an image processing flow on the infrared image to be processed according to the respective received image configuration parameters in step 103, and obtain an output image of the target image mode. For example, when the target image mode is the normal mode, the ARM processor in the Soc may obtain and send image configuration parameters (such as default parameters) corresponding to the normal mode to the corresponding hardware processing unit, for example, in the normal mode, the Soc may adopt single-frame input, and perform image processing of an image processing flow on the infrared image to be processed, which is input in a single frame, according to the default parameters by using the hardware processing unit corresponding to the normal mode, so as to obtain an output image in the normal mode; when the target image mode is the sky mode, the ARM processor in the Soc may obtain and send image configuration parameters corresponding to the sky mode to the corresponding hardware processing unit, such as modifying parameters of a data statistics part in an image enhancement process and a 3D image noise reduction process in default parameters, for example, the Soc may adopt single-frame input in the sky mode, and perform image processing of an image processing flow on an infrared image to be processed input in a single frame according to the partially modified default parameters by using the hardware processing unit corresponding to the sky mode, so as to obtain an output image in the sky mode.
Specifically, the ARM processor in this embodiment can give consideration to scheduling of different tasks, and can dynamically invoke hardware processing resources such as a DSP, an NUP, or a GPU to operate according to hardware processing resources and parameter configuration of the Soc, so that algorithm operation is sufficiently efficient, and the image processing flow demand for processing the entire infrared image in real time is satisfied by adopting optimization modes such as algorithm flow optimization, compiler optimization, SIMD (single instruction stream multiple data stream) instruction set optimization, cevedsp instruction set, and NEON multithreading.
In the embodiment of the invention, the one or more hardware processing units corresponding to the target image mode are utilized to perform image processing on the infrared image to be processed according to the image processing flow of the target image mode, so as to obtain the output image of the target image mode, and the imaging with higher complexity and better processing effect can be completed by utilizing different hardware resources in the Soc in a direct drive mode of the detector, so that the infrared imaging with low cost, high quality and high efficiency is realized on the basis of the Soc without adopting an FPGA (field programmable gate array), and the user experience is improved.
Corresponding to the above method embodiment, the embodiment of the present invention further provides an image processing apparatus for an infrared image, and an image processing apparatus for an infrared image described below and an image processing method for an infrared image described above may be referred to correspondingly.
Referring to fig. 6, fig. 6 is a block diagram of an image processing apparatus for infrared images according to an embodiment of the present invention. The device is applied to a system-on-chip and can comprise:
the image acquisition module 10 is configured to determine a current target image mode, and acquire an infrared image to be processed, which is acquired by the detector, according to the target image mode;
a hardware determining module 20, configured to determine one or more hardware processing units corresponding to the target image mode according to hardware resources in the system-on-chip, so as to split an image processing flow of the target image mode into the hardware processing units;
the image processing module 30 is configured to perform image processing on the infrared image to be processed according to an image processing flow of the target image mode by using one or more hardware processing units, and acquire an output image of the target image mode; wherein the image processing flow comprises at least one of image enhancement, non-uniformity correction and image noise reduction.
Optionally, when the image processing flow includes image enhancement, non-uniformity correction, and image noise reduction, and the plurality of hardware processing units corresponding to the target image mode include an ARM, a DSP, and an ISP, the image processing module 30 may include:
the correction submodule is used for carrying out non-uniformity correction on the infrared image to be processed by utilizing the ARM so as to obtain a corrected image;
the 2D noise reduction sub-module is used for carrying out image 2D noise reduction on the corrected image by utilizing an ISP (Internet service provider) to obtain a noise-reduced image;
the image enhancement submodule is used for carrying out image enhancement corresponding to a target image mode on the noise-reduced image by utilizing the DSP to obtain an enhanced image;
and the 3D noise reduction submodule is used for carrying out image 3D noise reduction on the enhanced image by utilizing the ISP to obtain an output image.
Optionally, when the target image mode is a wide dynamic mode, the image enhancement module may include:
the high-pass filtering unit is used for carrying out high-pass filtering on the noise-reduced images in the preset number to obtain high-frequency images in the preset number; the noise reduction image is an image corresponding to the infrared image to be processed, which is acquired by the detector and has different integration times of a preset number of frames;
the Gaussian filtering unit is used for carrying out Gaussian filtering on the high-frequency images to obtain a preset number of weighted images;
the low-pass filtering unit is used for carrying out low-pass filtering on the noise-reduced images to obtain a preset number of base layer images;
the detail acquiring unit is used for acquiring a preset number of detail images according to the noise-reduced image and the base layer image; wherein, the detail image is the difference between each noise reduction image and the corresponding base layer image;
and the fusion unit is used for fusing the base layer image and the detail image according to the weight image to obtain a frame of enhanced image.
Alternatively, when the preset number is 3, the fusion unit may be specifically configured to calculate an enhanced image by fusion (w1 f1+ w2 f2+ w3 f 3)/w; wherein fusion is an enhanced image, f1 ═ base1+ detail1, f2 ═ base2+ detail2, f3 ═ base3+ detail3, w ═ w1+ w2+ w3, w1, w2, and w3 are weighted images, base1, base2, and base3 are base layer images, and detail1, detail2, and detail3 are detail images.
Optionally, when the non-uniformity correction includes a streak non-uniformity correction, the syndrome module may include:
the fringe non-uniformity correction unit is used for performing fringe non-uniformity correction on the infrared image to be processed by utilizing the ARM to acquire a corrected image; the infrared image to be processed is an image which is output by the detector and is subjected to response consistency correction.
Optionally, when the target image mode is a wide dynamic mode, the apparatus may further include:
the detector configuration module is used for sending image frame output parameters corresponding to the wide dynamic mode to the detector; the image framing parameters include a preset number of different integration times.
Optionally, the apparatus may further include:
the parameter acquisition module is used for acquiring image configuration parameters corresponding to the target image mode;
the parameter configuration module is used for sending the image configuration parameters to the corresponding hardware processing units;
correspondingly, the image processing module 30 may be specifically configured to perform image processing of an image processing procedure on the infrared image to be processed according to the image configuration parameters by using one or more hardware processing units, so as to obtain an output image.
In this embodiment, in the embodiment of the present invention, the image processing module 30 performs image processing on the infrared image to be processed according to the image processing flow of the target image mode by using one or more hardware processing units corresponding to the target image mode, so as to obtain the output image of the target image mode, and can use a direct drive mode of the detector to complete imaging with high complexity and good processing effect by using different hardware resources in the Soc, so that under the condition of not using the FPGA, infrared imaging with low cost, high quality and high efficiency is realized based on the Soc, and user experience is improved.
Corresponding to the above method embodiment, the embodiment of the present invention further provides a system-on-chip, and a system-on-chip described below and an image processing method for an infrared image described above may be referred to in correspondence.
A system-on-chip comprising: a memory for storing a computer program; a processor for implementing the steps of the image processing method of the infrared image provided by the above embodiments when executing the computer program.
The system-on-chip provided by this embodiment may include a hardware processing unit, and the hardware processing unit may include at least one of an ARM, a DSP, a GPU, and an NPU.
Corresponding to the above method embodiment, the embodiment of the present invention further provides a computer readable storage medium, and a computer readable storage medium described below and an image processing method of an infrared image described above may be referred to in correspondence with each other.
A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for image processing of infrared images as provided by the above-mentioned method embodiments.
The computer-readable storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various other readable storage media capable of storing program codes.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device, the system on chip and the computer-readable storage medium disclosed in the embodiments correspond to the method disclosed in the embodiments, so that the description is simple, and the relevant points can be referred to the description of the method.
The image processing method, the image processing device, the system on chip and the computer readable storage medium for the infrared image provided by the invention are described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111355544.6A CN114049248A (en) | 2021-11-16 | 2021-11-16 | Image processing method, device, chip and storage medium for infrared image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111355544.6A CN114049248A (en) | 2021-11-16 | 2021-11-16 | Image processing method, device, chip and storage medium for infrared image |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114049248A true CN114049248A (en) | 2022-02-15 |
Family
ID=80209381
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111355544.6A Pending CN114049248A (en) | 2021-11-16 | 2021-11-16 | Image processing method, device, chip and storage medium for infrared image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114049248A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102801912A (en) * | 2012-07-17 | 2012-11-28 | 北京伽略电子系统技术有限公司 | Infrared image processing chip and processing method |
| CN106357987A (en) * | 2016-10-19 | 2017-01-25 | 浙江大华技术股份有限公司 | Exposure method and device |
| CN111492650A (en) * | 2018-02-09 | 2020-08-04 | 深圳开阳电子股份有限公司 | Image preprocessing method and device, image sensor interface, image processing method and device |
| CN112818732A (en) * | 2020-08-11 | 2021-05-18 | 腾讯科技（深圳）有限公司 | Image processing method and device, computer equipment and storage medium |
| US20230177713A1 (en) * | 2020-03-27 | 2023-06-08 | Sony Group Corporation | Information processing apparatus, information processing method, and program |
-
2021
- 2021-11-16 CN CN202111355544.6A patent/CN114049248A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102801912A (en) * | 2012-07-17 | 2012-11-28 | 北京伽略电子系统技术有限公司 | Infrared image processing chip and processing method |
| CN106357987A (en) * | 2016-10-19 | 2017-01-25 | 浙江大华技术股份有限公司 | Exposure method and device |
| CN111492650A (en) * | 2018-02-09 | 2020-08-04 | 深圳开阳电子股份有限公司 | Image preprocessing method and device, image sensor interface, image processing method and device |
| US20230177713A1 (en) * | 2020-03-27 | 2023-06-08 | Sony Group Corporation | Information processing apparatus, information processing method, and program |
| CN112818732A (en) * | 2020-08-11 | 2021-05-18 | 腾讯科技（深圳）有限公司 | Image processing method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI674804B (en) | Video dehazing device and method | |
| US9041834B2 (en) | Systems and methods for reducing noise in video streams | |
| TWI607409B (en) | Methods for enhancing images and apparatuses using the same | |
| US20140340486A1 (en) | Image processing system, image processing method, and image processing program | |
| Rouf et al. | Glare encoding of high dynamic range images | |
| CN112308797B (en) | Corner detection method and device, electronic equipment and readable storage medium | |
| CN103530896A (en) | Image compression and detail enhancement method for infrared image | |
| CN107135388A (en) | A Depth Extraction Method of Light Field Image | |
| CN107194946B (en) | FPGA-based infrared salient object detection method | |
| JP2016527611A (en) | System and method for single frame based super resolution interpolation for digital cameras | |
| CN119091379A (en) | A machine vision unit for smart community security management system | |
| Lapray et al. | Hdr-artist: High dynamic range advanced real-time imaging system | |
| Ye et al. | Lfienet: Light field image enhancement network by fusing exposures of lf-dslr image pairs | |
| Qi et al. | Deblurring neural radiance fields with event-driven bundle adjustment | |
| Shen et al. | Depth map enhancement method based on joint bilateral filter | |
| WO2025145962A1 (en) | Image processing method and apparatus, electronic device, and computer-readable storage medium | |
| Roomi et al. | A novel de-ghosting image fusion technique for multi-exposure, multi-focus images using guided image filtering | |
| CN114049248A (en) | Image processing method, device, chip and storage medium for infrared image | |
| Vanmali et al. | Multi-exposure image fusion for dynamic scenes without ghost effect | |
| Wang et al. | A bilateral filtering based ringing elimination approach for motion-blurred restoration image | |
| US10552970B2 (en) | Efficient guide filter for depth refinement | |
| Yu et al. | A novel Retinex based enhancement algorithm considering noise | |
| Alam et al. | Dynamic range and depth of field extension using camera array | |
| Petrescu et al. | Kinect depth inpainting in real time | |
| CN112347972A (en) | High-dynamic region-of-interest image processing method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |