CN117522681A - Real-time splicing method and system for UAV thermal infrared remote sensing images - Google Patents

Abstract

The invention relates to the field of unmanned aerial vehicle photography, and discloses a real-time splicing method and a real-time splicing system for thermal infrared remote sensing images of unmanned aerial vehicles. The method comprises the following steps: receiving an original thermal infrared image with pose information issued by an unmanned aerial vehicle; calculating an external azimuth element of the original thermal infrared image in a ground photographic coordinate system; carrying out orthorectification on the original thermal infrared image according to the external azimuth element to generate a thermal infrared orthographic image; adding a layer for recording the image height of the pixel point in the original thermal infrared image into the thermal infrared orthographic image; red heatSplicing the external orthographic image and the spliced image, wherein the gray value of the overlapping area of the thermal infrared orthographic image and the spliced image is based on the image height I _H The value of (2) is subjected to image fusion by adopting a gradual-in gradual-out method; and loading the updated spliced image into a ground control terminal GIS system, and superposing the spliced image with the existing satellite base map of the observation area for real-time display.

Description

Real-time splicing method and system for UAV thermal infrared remote sensing images

Technical field

The invention relates to the technical field of drone photography, and in particular to a real-time splicing method and system for thermal infrared remote sensing images of drones.

Background technique

Drones equipped with thermal infrared cameras can quickly achieve high-resolution thermal infrared remote sensing imaging of large areas. They have been widely used in professional fields such as fire emergency, field search and rescue, forest fire monitoring, etc., and can effectively reduce the personal injury caused by various emergencies. and property damage, etc. Subject to limitations such as thermal infrared camera resolution and flight altitude, thermal infrared remote sensing images need to be spliced to quickly achieve large-area temperature situation awareness. Currently, in various application scenarios, such as forest fire site survey, there is a strong demand for online real-time splicing of UAV thermal infrared remote sensing images. The large-area splicing method of UAV thermal infrared remote sensing images mainly collects photogrammetry or motion recovery structure methods, follows the image processing strategy of first registration and then splicing, and solves the image position and attitude by extracting the same-name features of the image overlapping areas. Then perform geometric correction and image stitching processing. Since UAV thermal infrared remote sensing images have low signal-to-noise ratio and contrast, feature matching is difficult, and hardware computing resources are consumed, resulting in online real-time splicing processing that is prone to failure.

Therefore, the technology has yet to be developed and improved.

Contents of the invention

The present invention provides a real-time splicing method and system for UAV thermal infrared remote sensing images, which is used to achieve thermal infrared remote sensing that is not limited by the extraction of features of the same name in surface scenes and images while requiring less computing resource support from the ground control end. Images are stitched online in real time.

The first aspect of the present invention provides a real-time splicing method of UAV thermal infrared remote sensing images. The real-time splicing method of UAV thermal infrared remote sensing images includes: calibrating the thermal infrared camera and the POS system. The POS system The system includes a GNSS receiver and IMU; receives the original thermal infrared image with pose element information sent by the drone; and based on the external parameter matrix of the thermal infrared camera relative to the imu coordinate system and the exposure time recorded in the pose element information image pose, calculate the external orientation elements of the original thermal infrared image in the ground photography coordinate system O _m ; calculate the original thermal infrared image according to the external orientation elements of the original thermal infrared image in the ground photography coordinate system O _m The image is orthorectified to generate a thermal infrared orthoimage; a layer for recording the image height I _H of each pixel in the original thermal infrared image is added to the generated thermal infrared orthoimage; when the generated thermal infrared orthoimage is When the thermal infrared orthoimage is not the first image, the generated thermal infrared orthoimage and the spliced image are spliced according to the geographical location, and the overlap between the generated thermal infrared orthoimage and the spliced image is The gray value of the area is fused using the fade-in and fade-out method according to the image height _I The existing satellite base map is overlaid and displayed in real time.

Preferably, the calibration of the thermal infrared camera and the thermal infrared camera and the POS system includes: calibrating the thermal infrared camera, and obtaining the focal length (f _x , f _y ) and principal point (c _x , ) of the thermal infrared camera. c _y ) and lens geometric distortion parameters (k ₁ , k ₂ , p ₁ , p ₂ , k ₃ ); synchronize the PPS signal received by the GNSS receiver to the clock of the IMU to synchronize the time of the IMU with the time of the GNSS receiver , and configure the thermal infrared camera to respond to the PPS trigger signal of the GNSS receiver so that the image captured by the thermal infrared camera is synchronized with the time of the GNSS receiver; obtain the external parameter matrix of the thermal infrared camera relative to the imu coordinate system.

Preferably, the method for obtaining the image pose at the exposure time recorded in the pose element information includes: after the thermal infrared camera image mounted on the drone is exposed, record the exposure time through the onboard control panel, and obtain the IMU the current timestamp of The image pose at the exposure moment is written into the Exif meta-information of the original thermal infrared image.

Preferably, the outer orientation elements of the original thermal infrared image in the ground photography coordinate system O _m include angular elements and line elements, and the angular elements are represented by a rotation matrix Represents, rotation matrix/> The calculation formula is as follows:

In the formula, m represents the ground photography coordinate system, E represents the geocentric three-dimensional rectangular coordinate system; g represents the navigation coordinate system, b represents the imu coordinate system, c represents the thermal infrared camera body coordinate system, and i represents the image space coordinate system, Represents the rotation matrix from i coordinate system to m coordinate system,/> Represents the rotation matrix from g coordinate system to e coordinate system,/> Represents the rotation matrix from b coordinate system to g coordinate system, /> Represents the rotation matrix from c coordinate system to b coordinate system, /> Represents the rotation matrix from i coordinate system to c coordinate system;

The line elements are represented by vectors Representation, vector/> The calculation formula is as follows:

In the formula, imu _E represents the origin of the imu coordinate system, m _E represents the coordinates of the origin of the ground photography coordinate system O _m in the geocentric three-dimensional rectangular coordinate system O _E , Indicates the coordinates of the origin of the thermal infrared camera coordinate system c in the imu coordinate system b.

Preferably, orthorectifying the original thermal infrared image according to the outer azimuth elements of the original thermal infrared image in the ground photography coordinate system _Om to generate a thermal infrared orthoimage includes: loading the observation area The incoming open DEM data set is projected into the ground photography coordinate system O _m through coordinate system conversion; based on the beam tracking positive calculation method, the actual four corners of the original thermal infrared image in the ground photography coordinate system O _m are calculated. The formula for geographical location (X _A , Y _A ), X _A and Y _A is:

In the formula, Z represents the average height of the DEM within the observation area, x and y represent the image pixel coordinates corrected by lens geometric distortion and image principal point offset, and f represents the average focal length (f _x , f _y ) of the thermal infrared camera. Values, x, y represent the image pixel coordinates corrected by lens geometric distortion and image principal point offset, f represents the average value of the focal length (f _x , f _y ) of the thermal infrared camera;

The beam tracking inverse calculation method is used to perform orthorectification processing on the original thermal infrared image, and a thermal infrared orthoimage of the original thermal infrared image in the ground photogrammetry coordinate system is generated. The inverse calculation formula of the pixel coordinates x and y is expressed as:

_In the formula, Z _A represents _the height _{values of} surface Offset and lens geometric distortion processing are used to obtain the actual pixel positions of the pixel coordinates x and y in the original thermal infrared image, and the radiation intensity values recorded at the actual pixel positions of the pixel coordinates x and y in the original thermal infrared image are filled in Described thermal infrared orthophoto.

Preferably, the fusion formula of the gray value Gray _{overlapping area} of the overlapping area of the generated thermal infrared orthophoto image and the spliced image is expressed as:

Preferably, the method further includes, when the generated thermal infrared orthoimage is the first image, loading the generated thermal infrared orthoimage into the ground control terminal GIS system as a spliced image and aligning it with the observation area. There is a satellite base map for overlay and real-time display.

The second aspect of the present invention provides a real-time splicing system for UAV thermal infrared remote sensing images, including a POS system, a thermal infrared camera, an FPGA chip, a microcontrol processor and a data memory. The POS system includes a GNSS receiver and an IMU, The GNSS receiver, the IMU, the thermal infrared camera, the FPGA chip and the data memory are respectively connected to the microcontrol processor, and the GNSS receiver, the IMU and the thermal infrared camera are respectively connected to the FPGA chip, the thermal infrared camera is connected to the data storage, the GNSS receiver is used to collect the PPS signal and the real-time position information of the thermal infrared camera, and send it to the FPGA chip, the The IMU is used to collect the real-time posture information of the thermal infrared camera and send it to the FPGA chip. The thermal infrared camera is used to take photos according to the photo control signal of the FPGA and return the exposure end information to the FPGA. chip, and sends the collected original thermal infrared image to the microcontrol processor and data memory; the FPGA chip is used to combine the PPS signal, the real-time position information of the thermal infrared camera and the real-time attitude information of the thermal infrared camera Perform pose data fusion processing to obtain pose fusion data, and send it to the microcontrol processor; the microcontrol processor is used to control and receive the GNSS receiver, the IMU, the thermal infrared camera, and the The data sent by the FPGA chip is used to perform image splicing processing based on the original thermal infrared image and the pose fusion data, and the processed data is stored in the data memory.

In the technical solution provided by the present invention, the aerial position and attitude of the UAV thermal infrared image at the time of exposure are obtained online through the POS system and spatio-temporal synchronous calibration technology, and orthorectification is completed while the original thermal infrared image is collected, and through the geographical splicing method With the fade-in and fade-out image fusion technology, real-time splicing of UAV thermal infrared images over a large area is carried out; this solution requires less computing resources, improves operational efficiency, and can effectively enhance UAV thermal infrared remote sensing technology for large areas. Timeliness of temperature situational awareness.

Description of drawings

Figure 1 is a flow chart of a real-time splicing method for UAV thermal infrared remote sensing images provided by an embodiment of the present invention;

Figure 2 is a structural block diagram of a real-time splicing system for UAV thermal infrared remote sensing images provided by an embodiment of the present invention.

Explanation of reference numbers:

10. GNSS receiver; 20. IMU; 30. Thermal infrared camera; 40. FPGA chip; 50. Microcontrol processor; 60. Data memory.

Detailed ways

Embodiments of the present invention provide a method and system for real-time splicing of UAV thermal infrared remote sensing images. The method is used to achieve thermal processing that is not limited by the extraction of features of the same name in surface scenes and images while requiring less computing resource support from the ground control end. Online real-time stitching of infrared remote sensing images.

The terms "first", "second", "third", "fourth", etc. (if present) in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects without necessarily using Used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. In addition, the terms "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusions, e.g., processes, methods, systems, products, or devices that comprise a series of steps or units and are not necessarily limited to those expressly listed. steps or units, but may include other steps or units not expressly listed or inherent to such processes, methods, products or apparatuses.

For ease of understanding, the specific process of the embodiment of the present invention is described below. Please refer to Figure 1. In the embodiment of the present invention, a real-time splicing method of UAV thermal infrared remote sensing images includes:

S101. Calibrate the thermal infrared camera and the POS system. The POS system includes a GNSS receiver and an IMU (Inertial Measurement Unit).

S102. Receive the original thermal infrared image with pose element information sent by the drone.

S103. Calculate the external orientation elements of the original thermal infrared image in the ground photography coordinate system O _m based on the external parameter matrix of the thermal infrared camera relative to the imu coordinate system and the image pose at the exposure time recorded in the pose element information.

S104. Perform orthorectification on the original thermal infrared image based on the external orientation elements of the original thermal infrared image in the ground photography coordinate system O _m to generate a thermal infrared orthoimage.

S105. Add a layer for recording the image height I _H of each pixel in the original thermal infrared image to the generated thermal infrared orthophoto.

S106. When the generated thermal infrared orthoimage is not the first image, splice the generated thermal infrared orthoimage and the spliced image according to the geographical location, and the overlapping area between the generated thermal infrared orthoimage and the spliced image The grayscale value of I uses the fade-in and fade-out method to perform image fusion according to the value of the image height I _H , and updates the spliced image.

Understandably, if the generated thermal infrared orthoimage is the first image, the generated thermal infrared orthoimage will be loaded into the ground control GIS system as a spliced image, and combined with the existing satellite base map of the observation area. Perform overlay real-time display.

S107. Load the updated spliced image into the ground control terminal GIS system, and overlay it with the existing satellite base map of the observation area for real-time display, for real-time perception of the temperature situation in the observation area.

In this embodiment, after completing the real-time splicing task of receiving the thermal infrared images of the drone, the ground system waits to receive the download of the image from the drone communication link. After receiving the new downloaded image, jump to S103 to continue the real-time splicing process; when receiving the image collection end flag, save the updated spliced image to a local file, the splicing task ends, and the drone follows the planned route Automatically return to home and land.

In this embodiment, in step S101, calibrating the thermal infrared camera and the thermal infrared camera and the POS system specifically includes: calibrating the thermal infrared camera, and obtaining the focal length (f _x , f _y ) and principal point of the thermal infrared camera. (c _x , c _y ) and lens geometric distortion parameters (k ₁ , k ₂ , p ₁ , p ₂ , k ₃ ); synchronize the PPS signal of the GNSS receiver to the clock of the IMU so that the time of the IMU is consistent with the time of the GNSS Synchronize and configure the thermal infrared camera to respond to the PPS trigger signal of the GNSS receiver so that the image captured by the thermal infrared camera is synchronized with the time of the GNSS receiver; obtain the external parameter matrix of the thermal infrared camera relative to the imu coordinate system.

Optionally, select a calibration plate with strong reflective ability for the thermal infrared band, and non-uniformly heat the calibration plate to complete the collection of multiple thermal infrared images from different angles. Use the "Zhang Zhengyou" method to calibrate the thermal infrared camera, and obtain the thermal infrared image. The focal length (f _x , f _y ), principal point (c _x , c _y ) and lens geometric distortion parameters (k ₁ , k ₂ , p ₁ , p ₂ , k ₃ ) of the infrared camera.

Understandably, the PPS (Pulse Per Second) signal is a precise pulse signal that is widely used in time synchronization and measurement. The PPS signal generates a pulse at a frequency of once per second. The PPS signal can provide a very precise time stamp, usually with an error of a few microseconds or less.

In this embodiment, the GNSS receiver receives the satellite PPS signal to obtain the timestamp, and the IMU generates acceleration and angular velocity data to ensure that its timestamp is synchronized with the GNSS receiver. At the same time, the thermal infrared camera is configured to respond to the PPS trigger signal to ensure that it is at a precise time point. Capture image. During data collection, the timestamps of each hardware are kept synchronized, and the image exposure time is stored and recorded by taking the average of the start and end exposure time.

Alternatively, use an outdoor control field with known coordinates and significant landmark points, and use a rigidly connected thermal infrared camera and POS system to collect thermal infrared images and IMU data. According to the basic principle that the image recovery trajectory and the IUM measurement trajectory should be consistent, Use the least squares optimization method to solve the external parameter matrix of the thermal infrared camera relative to the imu coordinate system

Optionally, when completing the calibration of the thermal infrared camera and the thermal infrared camera with the POS system, you also need to measure the lever arm value from the antenna phase center of the GNSS receiver to the origin mark of the imu coordinate system, and fill it in the acquisition control board in the card’s embedded software.

Optionally, before the drone takes off, in addition to completing the calibration of the thermal infrared camera and the thermal infrared camera and POS system, you also need to make preparations for the drone's pre-takeoff inspection. The preparation for the drone's pre-takeoff inspection includes: payload collection Configure the RTK signal source in the embedded software, and you can choose the network RTK mode or set up an RTK base station in the measurement area; then test the wireless communication between the payload acquisition terminal and the ground control terminal to ensure that the thermal infrared image can be downloaded normally; secondly, work on the POS system The status is confirmed to ensure that the time synchronization and pose collection information are normal; finally, the actual mapping resolution of the spliced image is set on the ground control terminal according to the actual pixel size GSD of the thermal infrared camera on the ground, where GSD is estimated by the following formula:

In the formula, u is the real size of the pixel, H is the relative altitude, and f is the physical focal length of the thermal infrared camera.

Furthermore, after calibrating the thermal infrared camera and POS system and inspecting the drone before taking off, based on the pre-planned route task, the ground control station automatically loaded the globally open DEM data set within the measurement area and controlled The UAV carries a thermal infrared camera and automatically takes off to start executing the designed route. In order to ensure 100% coverage of the measurement area and real-time splicing effect of the UAV thermal infrared image, the route design ensures that the image heading and side overlap rate is not less than 20%.

In this embodiment, in step S102, after the flight mission starts, the thermal infrared image real-time splicing function module is started in the ground control station, and the communication link monitors and receives the pose element information sent by the drone in real time. Raw thermal infrared image.

Optionally, the method of obtaining the image pose at the exposure moment recorded in the pose element information includes: after the thermal infrared camera image mounted on the drone is exposed, record the exposure moment through the airborne control panel and obtain the current time of the IMU. Stamp; when the current timestamp is greater than the exposure time, the two closest pose data before and after the exposure time are used to calculate the position and attitude of the thermal infrared image according to the linear interpolation method to obtain the image pose at the exposure time; the image at the exposure time is The pose is written into the Exif meta-information of the original thermal infrared image.

Specifically, the position and attitude are represented by geodetic latitude, geodetic longitude, ellipsoid height (B, L, H), and heading, pitch, and roll angles (heading, pitch, roll).

In this embodiment, in step S103, the origin of the coordinates in the ground photography coordinate system O _m is the longitude and latitude of the center point of the survey area and the height of the earth B ₀ , L ₀ , H ₀ , where the X and Y axes point to the true east and true north directions respectively. , the Z-axis direction points to the zenith according to the right-hand principle.

Optionally, the external orientation elements of the original thermal infrared image in the ground photography coordinate system O _m include angular elements and line elements, and the angular elements are represented by a rotation matrix Represents, rotation matrix/> The calculation formula is as follows:

In the formula, m represents the ground photography coordinate system, E represents the geocentric three-dimensional rectangular coordinate system; g represents the navigation coordinate system, b represents the imu coordinate system, c represents the thermal infrared camera body coordinate system, and i represents the image space coordinate system, Represents the rotation matrix from i coordinate system to m coordinate system, /> Represents the rotation matrix from g coordinate system to e coordinate system,/> Represents the rotation matrix from b coordinate system to g coordinate system,/> Represents the rotation matrix from c coordinate system to b coordinate system,/> Represents the rotation matrix from the i coordinate system to the c coordinate system.

Line elements use vectors Representation, vector/> The calculation formula is as follows:

In the formula, imu _E represents the origin of the imu coordinate system, m _E represents the coordinates of the origin of the ground photography coordinate system O _m in the geocentric three-dimensional rectangular coordinate system O _E , Represents the coordinates of the origin of the thermal infrared camera coordinate system c in the imu coordinate system b, Xs, Ys, Zs represent vectors/> of each position element.

In this embodiment, in step S104, the original thermal infrared image is orthorectified according to the outer azimuth elements of the original thermal infrared image in the ground photography coordinate system O _m to generate a thermal infrared orthoimage, including:

The open DEM data set loaded in the observation area is converted into the coordinate system and projected into the ground photography coordinate system O _m ; based on the beam tracking positive calculation method, the four corners of the original thermal infrared image are calculated in the ground photography coordinate system O _m The actual geographical location (X _A , Y _A ) in The pixel coordinates x and y are processed by principal point offset and lens geometric distortion to obtain the actual pixel positions of the pixel coordinates x and y in the original thermal infrared image, and the actual pixel positions of the pixel coordinates x and y in the original thermal infrared image are obtained. The radiation intensity values recorded at the location are filled into the corresponding thermal infrared orthophoto.

Among them, the formulas of X _A and Y _A are expressed as:

In the formula, represents the average height of the DEM within the observation area, x, y represents the image pixel coordinates corrected by lens geometric distortion and image principal point offset, f represents the average focal length (f _x , f _y ) of the thermal infrared camera, x, y represents the image pixel coordinates corrected by lens geometric distortion and image principal point offset, f represents the average of the focal length (f _x , f _y ) of the thermal infrared camera;

The inverse formula of pixel coordinates x and y is expressed as:

In the formula, Z _A represents the surface X _A and Y _A height values interpolated from DEM data, and f represents the average value of the focal length (f _x , f _y ) of the thermal infrared camera.

In this embodiment, in step S105, a layer for recording the image height I _H of each pixel in the original thermal infrared image is added to the generated thermal infrared orthophoto image for subsequent splicing and fusion. The smaller the image height I _H corresponding to the pixel in the thermal infrared orthophoto image, and the closer it is to the main image point in the original thermal infrared image, the smaller the imaging distortion, and the better the image quality.

The formula for the image height I _H of a certain pixel (x, y) in the original thermal infrared image is expressed as:

In the formula, (c _x , c _y ) represents the principal point of the thermal infrared camera.

In this embodiment, in step S106, when the image height of the spliced image is greater than the image height of the image to be spliced, the gray value of the overlapping area is equal to the gray value of the image to be spliced. When the image height of the spliced image is smaller than the image height to be spliced, When splicing the image height of the image, the gray value of the overlapping area is equal to the gray value of the spliced image. The fusion formula of the Gray overlapping _area of the generated thermal infrared orthophoto image and the overlapping area of the spliced image is expressed as:

In this embodiment, in step S107, the updated spliced image is loaded into the ground control end GIS system, and superimposed with the existing satellite base map of the observation area for real-time display.

In this embodiment, before loading the updated spliced image into the ground control GIS system, the updated spliced image can be appropriately feathered. The feathering process can create a smooth transition area between different images, so that The stitched images are more natural and continuous at the boundaries. This can avoid obvious edge breaks or mutations and improve the overall quality of the image.

During the image stitching process, artifacts or obvious transition lines may occur due to differences in brightness, color, etc. between different images. Feathering can reduce these artifacts, making the transition between images smoother and providing better visual effects.

This embodiment provides a real-time splicing method of UAV thermal infrared remote sensing images, which uses strict calibration of internal and external parameters of thermal infrared cameras in advance, multi-sensor high-precision time synchronization technology, and with the support of real-time fusion solution POS system, strictly Calculate the aerial position and attitude of the drone's thermal infrared image at the time of exposure, and then perform orthorectification on the original image transmitted to the ground. Finally, geometric splicing is performed based on the geographical location, and the image height is used to complete the image fade-in and fade-out fusion to achieve Online real-time stitching of UAV thermal infrared images. By adopting this method, there is no need to consume a large amount of local computing resources or use cloud computing. It can quickly complete the calculation of the aerial pose of the UAV thermal infrared image, and then use the geographical location and image height to achieve efficient splicing and high-quality fusion of adjacent images. , ensuring the timeliness, robustness and accuracy of the online splicing task of UAV thermal infrared images; at the same time, this method can greatly reduce the requirements for the overlap rate of adjacent images and improve the UAV's temperature situation awareness of the survey area. efficiency.

Please refer to Figure 2. The embodiment of the present invention also provides a real-time splicing system for UAV thermal infrared remote sensing images, including a POS system, a thermal infrared camera 30, an FPGA chip 40, a microcontrol processor 50 and a data memory 60. The POS system includes a GNSS receiver 10 and an IMU 20. The GNSS receiver 10, IMU 20, thermal infrared camera 30, FPGA chip 40 and data memory 60 are respectively connected to the micro control processor 50. The GNSS receiver 10, IMU 20 and thermal infrared camera 30 are respectively connected to the micro control processor 50. Connected to the FPGA chip 40 , the thermal infrared camera 30 is connected to the data storage 60 , the GNSS receiver 10 is used to collect PPS signals and real-time position information of the thermal infrared camera 30 , and sends them to the FPGA chip 40 , and the IMU 20 is used to collect the thermal infrared camera 30 The real-time posture information is sent to the FPGA chip 40. The thermal infrared camera 30 is used to perform photography according to the photographing control signal of the FPGA, return the exposure end information to the FPGA chip 40, and send the collected original thermal infrared image to the microcomputer. Control processor 50 and data memory 60; FPGA chip 40 is used to perform pose data fusion processing on PPS signals, real-time position information of thermal infrared camera 30 and real-time attitude information of thermal infrared camera 30 to obtain pose fusion data, and send To the microcontrol processor 50; the microcontroller 50 is used to control and receive data sent by the GNSS receiver 10, IMU20, thermal infrared camera 30, and FPGA chip 40, and perform image stitching based on the original thermal infrared image and pose fusion data. process and store the processed data in data storage 60.

The splicing system of this embodiment uses the POS system and spatio-temporal synchronization calibration technology to online obtain the aerial position and attitude of the UAV thermal infrared image at the time of exposure, completes orthorectification while collecting the original thermal infrared image, and uses the geographical splicing method and gradient The progressive image fusion technology enables real-time splicing of large-area thermal infrared images from drones; the system requires less computing resources, improves operational efficiency, and can effectively enhance the temperature situation of large areas using drone thermal infrared remote sensing technology. Perceived timeliness.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described systems, devices, and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code. .

As mentioned above, the above embodiments are only used to illustrate the technical solution of the present invention, but not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the foregoing. The technical solutions described in each embodiment may be modified, or some of the technical features may be equivalently replaced; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims (8)

1. The unmanned aerial vehicle thermal infrared remote sensing image real-time splicing method is characterized by comprising the following steps of:

checking the thermal infrared camera and the thermal infrared camera with a POS system, wherein the POS system comprises a GNSS receiver and an IMU;

receiving an original thermal infrared image with pose information issued by an unmanned aerial vehicle;

calculating the original thermal infrared image in a ground photographing coordinate system O according to the external parameter matrix of the thermal infrared camera relative to an imu coordinate system and the image pose of the exposure time recorded in pose meta information _m An external orientation element of (a);

photographing a coordinate system O on the ground according to the original thermal infrared image _m The original thermal infrared image is subjected to orthorectification by the external orientation element in the process, so that a thermal infrared orthographic image is generated;

increasing the image height I in the original thermal infrared image for recording each pixel in the generated thermal infrared orthographic image _H Is a layer of (2);

when the generated thermal infrared orthographic image is not the first image, the generated thermal infrared orthographic image and the generated thermal infrared orthographic image are combined according to the geographic positionSplicing the spliced images, wherein the gray value of the overlapping area of the generated thermal infrared orthographic image and the spliced image is based on the image height I _H The value of (2) adopts a gradual-in gradual-out method to carry out image fusion, and updates the spliced image;

and loading the updated spliced image into a ground control terminal GIS system, and superposing the spliced image with the existing satellite base map of the observation area for real-time display.

2. The method for real-time stitching of thermal infrared remote sensing images of an unmanned aerial vehicle according to claim 1, wherein the calibrating of the thermal infrared camera and the thermal infrared camera with the POS system comprises:

calibrating a thermal infrared camera to obtain a focal length (f) of the thermal infrared camera _x ,f _y ) Principal point (c) _x ,c _y ) And lens geometric distortion parameter (k) ₁ ,k ₂ ,p ₁ ,p ₂ ,k ₃ )；

Synchronizing PPS signals received by the GNSS receiver to a clock of the IMU to synchronize time of the IMU with time of the GNSS receiver, and configuring the thermal infrared camera to respond to PPS trigger signals of the GNSS receiver to synchronize images captured by the thermal infrared camera with time of the GNSS receiver;

and obtaining an extrinsic matrix of the thermal infrared camera relative to an imu coordinate system.

3. The method for real-time splicing of thermal infrared remote sensing images of an unmanned aerial vehicle according to claim 1, wherein the method for acquiring the image pose of the exposure time recorded in the pose meta-information comprises the following steps:

after the thermal infrared camera image carried by the unmanned aerial vehicle is exposed, recording the exposure time through an onboard control board;

acquiring a current time stamp of the IMU;

when the current time stamp is larger than the exposure time, calculating the position and the posture of the thermal infrared image according to a linear interpolation method by adopting two posture data which are closest to each other before and after the exposure time, and obtaining the image posture at the exposure time;

and writing the image pose at the exposure time into Exif meta information of the original thermal infrared image.

4. The method for real-time stitching of thermal infrared remote sensing images of unmanned aerial vehicle according to claim 3, wherein the original thermal infrared images are in a ground photographic coordinate system O _m The external azimuth element in (a) comprises an angle element and a line element, and the angle element is used for a rotation matrixRepresentation, rotation matrix->The calculation formula of (2) is as follows:

wherein m represents a ground photographing coordinate system, E represents a geocentric three-dimensional rectangular coordinate system; g represents a navigation coordinate system, b represents an imu coordinate system, c represents a thermal infrared camera body coordinate system, i represents an image space coordinate system,rotation matrix representing i coordinate system to m coordinate system,/->Rotation matrix representing g-coordinate system to e-coordinate system,/->Representing a rotation matrix from the b-coordinate system to the g-coordinate system,rotation matrix representing c-coordinate system to b-coordinate system,/->A rotation matrix representing the i coordinate system to the c coordinate system;

the line element vectorRepresenting vector->The calculation formula of (2) is as follows:

in the formula, imu _E Represents the origin of imu coordinate system, m _E Representing the ground photographic coordinate system O _m Origin is at three-dimensional rectangular coordinate system O of geocenter _E Is used to determine the coordinates of the coordinate system,representing the coordinates of the origin of the thermal infrared camera coordinate system c in the imu coordinate system b.

5. The method for real-time stitching of thermal infrared remote sensing images of an unmanned aerial vehicle according to claim 4, wherein said original thermal infrared images are stitched in a ground photographic coordinate system O _m The method comprises the steps of carrying out orthorectification on the original thermal infrared image by using the external orientation element to generate a thermal infrared orthographic image, and comprising the following steps:

the open DEM data set loaded in the observation area is projected to a ground photographic coordinate system O through coordinate system conversion _m In (a) and (b);

based on a beam tracking calculation method, the four-corner on-ground photographing coordinate system O of the original thermal infrared image is calculated _m Is the actual geographic location (X _A ,Y _A ),X _A And Y _A The formula of (2) is:

in the method, in the process of the invention,represents the average height of DEM in the observation area, x, y represents the image pixel coordinates corrected by lens geometric distortion correction and image principal point offset correction, f represents the focal length (f _x ,f _y ) X, y represent the pixel coordinates of the image corrected by the lens geometric distortion correction and the principal point offset correction, f represents the focal length (f _x ,f _y ) Average value of (2);

carrying out orthorectification treatment on the original thermal infrared image by adopting a beam tracking back calculation method to generate a thermal infrared orthographic image of the original thermal infrared image in a ground photogrammetry coordinate system, wherein a back calculation formula of pixel coordinates x and y is expressed as follows:

wherein Z is _A Representing the surface X interpolated from DEM data _A 、Y _A Height value, f denotes focal length of thermal infrared camera (f _x ,f _y ) Average value of (2);

and performing image principal point offset and lens geometric distortion treatment on the pixel coordinates x and y to obtain actual pixel positions of the pixel coordinates x and y in the original thermal infrared image, and filling radiation intensity values recorded by the actual pixel positions of the pixel coordinates x and y in the original thermal infrared image into the thermal infrared orthographic image.

6. The method for real-time stitching of thermal infrared remote sensing images of an unmanned aerial vehicle according to claim 1, wherein the Gray value Gray of the overlapping area of the generated thermal infrared orthographic image and the stitched image _{Overlapping region} Is expressed as:

7. the method for real-time stitching of thermal infrared remote sensing images of an unmanned aerial vehicle according to claim 1, further comprising loading the generated thermal infrared orthographic images as stitched images into a ground control terminal GIS system when the generated thermal infrared orthographic images are first images, and displaying the stitched images in real time by overlapping with an existing satellite base map of an observation area.

8. The unmanned aerial vehicle thermal infrared remote sensing image real-time splicing system is characterized by comprising a POS system, a thermal infrared camera, an FPGA chip, a micro control processor and a data memory, wherein the POS system comprises a GNSS receiver and an IMU, the GNSS receiver, the IMU, the thermal infrared camera, the FPGA chip and the data memory are respectively connected with the micro control processor, the GNSS receiver, the IMU and the thermal infrared camera are respectively connected with the FPGA chip, the thermal infrared camera is connected with the data memory, the GNSS receiver is used for acquiring PPS signals and real-time position information of the thermal infrared camera and sending the PPS signals and the real-time position information of the thermal infrared camera to the FPGA chip, the IMU is used for acquiring real-time attitude information of the thermal infrared camera and sending the real-time attitude information to the FPGA chip, and the thermal infrared camera is used for executing photographing according to photographing control signals of the FPGA and returning exposure end information to the FPGA chip and sending acquired original thermal infrared images to the micro control processor and the data memory; the FPGA chip is used for carrying out pose data fusion processing on the PPS signal, the real-time position information of the thermal infrared camera and the real-time pose information of the thermal infrared camera to obtain pose fusion data, and sending the pose fusion data to the micro-control processor; the micro-control processor is used for controlling and receiving data sent by the GNSS receiver, the IMU, the thermal infrared camera and the FPGA chip, performing image splicing processing according to the original thermal infrared image and the pose fusion data, and storing the processed data in the data memory.