CN119334892A - Hyperspectral oblique photography system and method based on unmanned aerial vehicle and ground agricultural machinery - Google Patents

Abstract

The present invention discloses a hyperspectral oblique photography method based on unmanned aerial vehicles and ground agricultural machinery, which uses unmanned aerial vehicles to obtain aerial data of farmland, uses agricultural machinery to continuously and in real time obtain ground data, and obtains data during the operation of agricultural machinery, without the need to invest additional time and manpower, and uses air-ground collaboration to integrate the advantages of air and ground platforms, so as to achieve high-precision, high-spectral resolution, and multi-angle comprehensive acquisition of farm remote sensing data, improve the spatial and spectral resolution of data, meet the data needs of precision agriculture, and use aerial oblique photography to obtain multi-viewing information of farmland to establish a high-precision three-dimensional model of farmland. The present invention can solve the problems of insufficient data acquisition accuracy and timeliness in the prior art, significantly improve the accuracy of remote sensing data acquisition in agricultural production, and is widely used in crop health monitoring, pest control, soil management and other fields.

Description

Hyperspectral oblique photography system and hyperspectral oblique photography method based on unmanned aerial vehicle and ground agricultural machinery

Technical Field

The invention belongs to the fields of precise agriculture, remote sensing technology, agricultural information technology and unmanned aerial vehicle application, and particularly relates to a hyperspectral oblique photography method based on an unmanned aerial vehicle and a ground agricultural machine.

Background

With the development of agricultural automation and informatization, precise agriculture has become a major direction of future agriculture. Accurate agriculture requires acquisition of high resolution crop and soil information for fine management and decision making. However, the existing remote sensing technology mainly depends on satellites or aviation platforms, is limited by spatial resolution and time resolution, and is difficult to meet the high-precision requirement of crop monitoring. Unmanned aerial vehicle is used as a low-cost and high-flexibility platform, is widely applied to agricultural data acquisition, but only depends on unmanned aerial vehicle to operate, and has certain limitations, such as difficulty in considering data precision and time continuity, and the following defects still exist when a single unmanned aerial vehicle platform acquires surface information:

1. The data precision is limited, the hyperspectral data acquired by the unmanned aerial vehicle are limited by the flying height and the performance of the sensor, and the space and the spectral resolution are difficult to be compatible.

2. The unmanned aerial vehicle is greatly influenced by weather, and the flight of the unmanned aerial vehicle is restricted by weather conditions, so that the continuity and the stability are poor.

3. The data processing is complex, the hyperspectral image data volume is large, the processing and analysis complexity is high, and high-performance computing equipment with heavier quality cannot be integrated.

Disclosure of Invention

The hyperspectral oblique photographing method based on the unmanned aerial vehicle and the ground agricultural machinery combines the aerial data of the unmanned aerial vehicle and the data of the ground agricultural machinery, utilizes the complementarity of the aerial data and the data of the ground agricultural machinery, effectively improves the precision and timeliness of remote sensing data, meets the hyperspectral and three-dimensional information requirements in precise agriculture, and provides a high-precision data acquisition and analysis tool for agricultural production and environment monitoring.

The technical scheme adopted by the invention is as follows:

A hyperspectral oblique photography method based on unmanned aerial vehicle and ground agricultural machinery comprises the following steps: S1, setting the synchronism of an unmanned aerial vehicle and an agricultural machine;

the unmanned aerial vehicle and the agricultural machine adopt a unified time synchronization mechanism, so that the aerial data collected by the unmanned aerial vehicle and the ground data collected by the agricultural machine have the same time standard.

In order to enable the aerial data collected by the unmanned aerial vehicle and the ground data collected by the agricultural machinery to have the same time standard, the time synchronization mechanism which can be adopted by the time synchronization mechanism comprises a GNSS time service mechanism, a network clock protocol mechanism, a timing signal trigger mechanism or a ground control station time unified instruction mechanism.

The GNSS timing mechanism is characterized in that GNSS modules supporting high-precision time synchronization are respectively installed on the unmanned aerial vehicle and the agricultural machine by using a global navigation satellite system GNSS (GlobalNavigationSatelliteSystem) such as a GPS (global positioning system), beidou and the like. The time signals provided by the GNSS satellites are accurate and uniform, and the time synchronization of the unmanned aerial vehicle and the agricultural machine can be ensured. And each time the GNSS modules on the unmanned aerial vehicle and the agricultural machine receive the same satellite time service signals, the unmanned aerial vehicle and the agricultural machine can start to synchronously acquire data under the time reference.

The network clock protocol mechanism is to establish a wireless network between the unmanned aerial vehicle and the agricultural machinery by adopting a network time protocol NTP or a precision time protocol PTP, wherein the unmanned aerial vehicle and the agricultural machinery are in the same network, a network time server provides a time reference, and the network time server performs time synchronization by using NTP (Network Time Protocol) or PTP (PrecisionTime Protocol). The NTP can keep the consistency of the unmanned aerial vehicle or agricultural machinery time at the millisecond level through the network time server, and the PTP has higher synchronous precision (microsecond level) and is suitable for scenes with higher requirements on the time consistency.

The timing signal triggering mechanism is to use a special timing signal generator, a receiving module is arranged on the unmanned aerial vehicle and the agricultural machinery, and the timing signal generator sends synchronous signals (such as radio pulses, wireless beacons and the like) at fixed frequency. And after receiving the signals, the unmanned aerial vehicle and the agricultural machine simultaneously start to collect data and uniformly record the collection start time, and keep synchronization on the basis. The synchronization mechanism may set a periodic trigger to keep the acquisition time points consistent.

The ground control station time unified instruction mechanism is to use the ground control station as a unified control center to monitor and control the synchronism of the unmanned aerial vehicle and the agricultural machinery in real time. The ground control station sends a synchronous command to the unmanned aerial vehicle and the agricultural machinery through a wireless network, so that the unmanned aerial vehicle and the agricultural machinery start data acquisition when receiving the command. And the synchronization mechanism can be integrated with the GNSS time service mechanism and the network clock protocol mechanism for application, so that the synchronism of a plurality of devices is ensured.

Of course, a time stamp alignment and post-processing synchronization mechanism can also be used, and alignment is performed according to the time stamp records of each device in the data post-processing stage. The above-mentioned several synchronization mechanisms are generally real-time synchronization mechanisms, and the time stamp alignment and post-processing mechanisms are applicable to the situation that real-time is not required, and if real-time synchronization is difficult to realize, the time of ground and air data can be matched through time stamp alignment during post-processing of the data.

By adopting the synchronization mechanism, the unmanned aerial vehicle and the agricultural machinery can share the same time reference in the data acquisition process, so that high-precision synchronization of aerial and ground data is realized, and reliable data support is provided for multi-source data fusion and farmland three-dimensional modeling.

S2, acquiring aerial data by the unmanned aerial vehicle;

the aerial data comprise aerial hyperspectral image data of farmlands, aerial inclined image data of farmlands, unmanned aerial vehicle position data and unmanned aerial vehicle attitude data;

The unmanned aerial vehicle is provided with a positioning module, an attitude sensor, a hyperspectral camera and an oblique photographic camera, and flies according to set conditions, wherein the set conditions comprise a flight path, a flight height, a flight speed, a route interval, a hyperspectral camera oblique angle and an oblique photographic camera oblique angle.

The unmanned aerial vehicle can be a multi-rotor or fixed wing, is provided with a high-precision positioning system and a gesture control system, wherein the positioning system uses a GPS system or an RTK system, and adopts a Real-TIME KINEMATIC (Real-time dynamic) carrier phase difference technology;

the hyperspectral camera is used for acquiring hyperspectral image data of farmland, and can be hyperspectral image data of single crops, hyperspectral image data of single soil and hyperspectral image data of both crops and soil;

The oblique photography camera is used for acquiring oblique photography image data of farmlands;

S3, acquiring ground data by the agricultural machinery;

the ground data comprise ground hyperspectral image data, agricultural machinery position data and agricultural machinery posture data;

the agricultural machinery is provided with a positioning module, an attitude sensor and a hyperspectral camera;

the method comprises the steps of continuously acquiring ground hyperspectral image data, agricultural machinery position data and agricultural machinery posture data in the farmland operation process, wherein the ground hyperspectral image data can be soil hyperspectral image data, crop hyperspectral image data and hyperspectral image data of soil and crops, and can be a tractor, a seeder or a harvester, and has the advantages of stable work, being close to the ground and convenient to install high-performance computing equipment;

S4, data preprocessing;

Preprocessing the aerial hyperspectral image data;

Performing radiation correction and atmosphere correction on the aerial hyperspectral image data;

carrying out geometric correction on the aerial hyperspectral image data subjected to radiation correction and atmosphere correction according to the unmanned aerial vehicle position data and the unmanned aerial vehicle posture data;

Preprocessing the aerial inclined image data;

Geometrically correcting the air inclined image data of the farmland according to the unmanned aerial vehicle position data and the unmanned aerial vehicle attitude data;

preprocessing ground hyperspectral image data;

performing radiation correction and atmosphere correction on the ground hyperspectral image data;

the ground hyperspectral image data after the radiation correction and the atmosphere correction are geometrically corrected according to the agricultural machinery position data and the agricultural machinery posture data;

s5, carrying out data fusion on the ground hyperspectral image data and the aerial hyperspectral image data;

The ground hyperspectral image data after pretreatment is spatially matched with the air hyperspectral image data after pretreatment.

The preprocessed ground hyperspectral image data and the aerial hyperspectral image data have consistent spectral response, and the space matching ensures the accurate alignment of the ground hyperspectral image data and the aerial hyperspectral image data in space, so that a foundation is provided for subsequent data fusion.

The specific steps of space matching are as follows:

unified coordinate system and geometric correction:

The ground hyperspectral image data and the aerial hyperspectral image data are converted into the same geographic coordinate system (such as WGS84 coordinate system) by using GNSS and IMU data so as to align the geographic positions of the images.

And then carrying out geometric correction on the image data in the same geographic coordinate system to ensure that the spatial geometric deformation of the inclined image is corrected, so that the image accurately corresponds to the ground coordinate.

Feature point detection and matching:

The key geographic feature points are respectively extracted from the ground hyperspectral image data and the air hyperspectral image data through a feature extraction algorithm, and the key address feature points generally comprise obvious landmarks, road intersection points, ridges and the like, and can be clearly identified in the ground and air images.

The adopted feature extraction algorithm can be an algorithm SIFT (Scale-INVARIANT FEATURE TRANSFORM) for detecting local features, a feature algorithm SURF (SpeedUp Features) with robustness of an acceleration edition or an algorithm ORB (Oriented FAST and Rotated BRIEF) for extracting and describing feature points rapidly;

And matching the key geographic feature points, and establishing spatial correlation of ground and aerial images through matched point pairs to provide a reference for the next spatial registration.

Image resampling and spatial registration:

The ground image is resampled to a spatial scale and resolution consistent with the aerial image using a geometric transformation (e.g., affine transformation or perspective transformation) based on the matching key geographic feature points, such that the ground image is geographically aligned with the aerial image.

Before spatial registration, stereo matching or point cloud generation technology (such as Structure-from-Motion, sfM) is required to be used for the aerial image data obtained by oblique photography, multi-view image fusion is carried out, global optimization reconstruction is carried out to obtain three-dimensional point cloud after multi-view image fusion, and accurate alignment of ground images and aerial images on spatial coordinates is ensured.

For areas with irregular terrain or large differences in elevation, digital elevation model DEM (Digital Elevation Model) or digital surface model DSM (DigitalSurface Model) may be used for spatial registration to align the ground image with the aerial image over the terrain;

the aligned images are resampled to be adjusted to be consistent in resolution, accurate correspondence on the image space is ensured, and a foundation is laid for spectrum fusion.

Verification and error correction:

By verifying the alignment accuracy of the typical areas (e.g., landmark points) in the images, it is ensured that the ground and aerial images are spatially exactly matched. If deviation exists, the characteristic points and the registration parameters can be further adjusted. And calculating the precision after registration by using an error analysis method, and ensuring that the space matching meets the requirements.

Through the steps, accurate spatial matching of ground and aerial hyperspectral images can be realized, and a stable foundation is provided for further fusion of spectrum data and hyperspectral feature construction of a farmland three-dimensional model.

And after space matching, carrying out spectrum fusion on the ground hyperspectral image data and the aerial hyperspectral image data to obtain fused hyperspectral image data.

The specific steps of spectrum fusion are as follows:

Spectral normalization:

since the spectra of ground and aerial images may vary depending on the acquisition height and sensor characteristics, the images are first spectrally normalized to make the spectral intensities uniform.

Normalization may be performed using radiation correction and atmospheric correction methods (e.g., FLAASH or QUAC) to reduce spectral differences between ground and aerial data.

Spectral band matching and resampling:

the ground and aerial image data are aligned to a uniform spectral band. If the data spectrum wave band distribution of the two groups of images is different, the inconsistent wave bands can be interpolated or resampled, so that the two images correspond to each other on the same wave band.

And processing the data of different wave bands by adopting an interpolation method (such as linear interpolation and spline interpolation) to ensure that all wave band information can be matched to the same spectrum scale.

Spectral weighted fusion:

And aiming at the overlapping area after space matching, adopting a weighted fusion strategy to carry out weighted synthesis on the spectral details of the ground image and the whole information of the aerial image. By setting the weight parameters, the spectrum information near the ground is more prominent in detail, and the aerial image occupies a larger proportion in macroscopically.

Different weights can be set according to the spatial resolution and visual difference of the images, and the ground images generally provide high-resolution details, so the weights are slightly high, the aerial images provide wide-area information, and the weights are slightly low. The fused spectrum information has the characteristics of both global visual angle and detail information.

Multiscale fusion:

And adopting a multi-scale fusion method (such as wavelet transformation or pyramid fusion) to fuse different scale information of the ground image and the aerial image. The wavelet transformation can decompose different scale information of the image, integrate high-frequency detail information of the ground image with low-frequency structure information of the aerial image, so that the details of the ground image are highlighted in a high-frequency part, and the integral structure of the aerial image is reserved in a low-frequency part.

And reconstructing after multi-scale fusion to generate complete hyperspectral image data containing high-frequency and low-frequency information, and retaining the details and integral information of the two.

Spectral smoothing and detail optimization:

And carrying out smoothing treatment on the fused spectrum data to reduce the discontinuity between spectrums and ensure the smooth transition of the spectrum data in the fused image.

The smoothing process uses a spectral smoothing algorithm (such as Gaussian smoothing or moving average) to process the spectral data, so that the spectral continuity of the image is enhanced, the naturalness of the spectral information is improved, and the spectral information is more continuous and real.

Outputting fused hyperspectral image data:

And storing the fused hyperspectral image data into a multiband image file (such as GeoTIFF format or HDF format) or a hyperspectral image data format, so as to ensure that each pixel contains complete spectral information.

The output fusion data can be directly used for further analysis (such as three-dimensional modeling or vegetation index calculation) and can also be used for visual display of hyperspectral images.

Through the spectrum fusion steps, fused hyperspectral image data with continuous space and rich spectrum can be generated. The image data not only maintains the detailed information of the ground image, but also combines the whole visual angle of the aerial image, and is suitable for the fine application of agricultural monitoring, crop growth analysis and the like.

The ground hyperspectral image data are fine and close, the defects of the aerial hyperspectral image data can be corrected and supplemented, and the spectral resolution and the spatial precision of the whole data are improved;

s6, constructing a farmland three-dimensional model with spectral characteristics;

The three-dimensional model of the farmland is constructed by using the preprocessed air inclined image data, which are collected multi-view images of the unmanned aerial vehicle for obliquely shooting the farmland from different angles and different heights, and can be generally realized by a multi-view three-dimensional reconstruction technology, so that the coverage range and the reconstruction precision are ensured. And radiation correction, geometric correction and atmospheric correction are performed on the aerial inclined image data during preprocessing, so that illumination difference, perspective deformation and atmospheric interference are removed, and the image data is more accurate.

The method specifically comprises the following steps:

Feature point detection and matching:

Feature extraction algorithms (e.g., SIFT, SURF, or ORB) are used to detect feature points in each image. These feature points should be consistent at different viewing angles to facilitate matching.

And matching the characteristic points in the multi-view images to find out the corresponding points in different images. The matching result forms an initial spatial geometrical relationship and provides a basis for three-dimensional reconstruction.

Structural-from-Motion (SfM):

A structural reconstruction algorithm (such as Structure-from-Motion) is used to generate a camera pose and an initial sparse point cloud based on the feature point matching results. SfM builds a sparse three-dimensional point cloud of a scene by solving for camera outliers (pose and position). The sparse point cloud generated at this stage reflects the basic spatial layout of the farmland, but the resolution is lower and the details are less.

Generating a dense point cloud:

On the basis of sparse point clouds, dense matching algorithms (such as Multi-View Stereo, MVS) are used to generate high-density point clouds. MVS restores more points to three-dimensional space based on disparities between multi-view images. The generated dense point cloud has higher resolution and rich details, and can accurately reflect fluctuation of farmland topography and surface morphology of crops.

Gridding and surface reconstruction:

And performing triangular meshing (such as Delaunay triangulation) on the dense point cloud data to generate a farmland three-dimensional grid model, wherein the meshing step is used for connecting the point cloud into a surface and providing a surface structure for the three-dimensional model.

And after gridding, carrying out surface smoothing and optimizing treatment to eliminate noise points or irregular surfaces, so that the surface of the model is smoother and more real.

Mapping the fused hyperspectral image data to a farmland three-dimensional model to obtain a farmland three-dimensional model with spectral characteristics;

The operation steps are as follows:

three-dimensional model and spatial calibration of fused hyperspectral image data:

The three-dimensional model and the coordinate system of the fused hyperspectral image data are aligned, and consistent geographic coordinates can be provided for the image and the three-dimensional model through GNSS and IMU data, so that the fused hyperspectral image data are matched with the spatial position of the farmland three-dimensional model.

If the three-dimensional model uses multi-view reconstruction (such as those generated by SfM and MVS), it is ensured that its position, scale and orientation are consistent with the hyperspectral image data.

Feature point matching and image projection:

Common geographic feature points (such as landmarks, road intersections and the like) are found in the farmland three-dimensional model and the fused hyperspectral image data, and feature matching algorithms (such as SIFT, SURF and the like) are used for registering the feature points so as to ensure that the fused hyperspectral image data and the three-dimensional model are accurately aligned.

And projecting the fused hyperspectral image data onto the surface of the three-dimensional model by using a projection transformation algorithm, so that each three-dimensional grid or point cloud point can correspond to a pixel in the fused hyperspectral image data.

And (3) mapping the fused hyperspectral image data to the surface of the farmland three-dimensional model:

and mapping each band information fused with the hyperspectral image data onto the surface texture of the farmland three-dimensional model. Typically, each band fused with hyperspectral image data represents a different spectral feature, and these band data are mapped hierarchically as texture information to each surface grid of the three-dimensional model of the farmland.

And directly mapping the pixel values fused with the hyperspectral image data to the corresponding surface positions of the three-dimensional model by using an image projection or ray projection technology, and generating a texture image layer containing spectral information.

Spectral data interpolation and weighted fusion:

For the part which is not directly aligned to the farmland three-dimensional model grid in the fused hyperspectral image data, an interpolation algorithm (such as bilinear interpolation or spline interpolation) can be used for filling the unaligned part, so that each grid point is ensured to have complete spectral data.

And a weighted fusion strategy can be used in the overlapped area to fuse the hyperspectral data of a plurality of visual angles to the surface of the model, so that the accuracy and the continuity of the spectrum information of the model are enhanced.

Multiband texture generation:

And layering the data of each wave band on the texture of the farmland three-dimensional model to generate a texture image containing multiple wave bands. Thus, a three-dimensional model with hyperspectral characteristics can be generated, and analysis and observation under different spectral bands are facilitated.

The surface unit (such as triangle patch or point) of each farmland three-dimensional model contains multi-band spectrum data, and can be used for subsequent vegetation health analysis, crop condition evaluation and other applications.

Model optimization and spectral feature extraction:

and optimizing the farmland three-dimensional model with the spectrum information to ensure that the hyperspectral data has smooth transition on the surface of the farmland three-dimensional model, and avoid abrupt change or discontinuity.

And further calculating spectral indexes (such as NDVI, NDRE and the like) and mapping the spectral indexes onto a farmland three-dimensional model, so that the farmland three-dimensional model presents spectral characteristic information of farmland, and the growth condition and health state of crops can be conveniently identified.

Outputting a three-dimensional model with spectral features:

the finally generated three-dimensional model with spectral features is exported into a format (such as PLY, OBJ, etc.) supporting multi-band data for subsequent further investigation in three-dimensional visualization software or analysis platforms.

The output model can be directly used for agricultural monitoring and accurate management, and different crop areas can be deeply analyzed by observing spectral characteristic information of the three-dimensional model.

Through the hyperspectral data mapping, the farmland three-dimensional model has accurate spectral characteristics, can support multi-level farmland information analysis, and is convenient for the fine evaluation and management of crop conditions.

As a preferable scheme of the invention, the farmland three-dimensional model with spectral characteristics can be applied to crop growth assessment, pest and disease damage monitoring and soil health analysis.

As a preferable scheme, when the soil health condition analysis is carried out, the aerial hyperspectral image data of the farmland are farmland soil surface layer spectrum data, the ground hyperspectral image data are ground soil spectrum data, the agricultural machinery is obtained when fertilization or sowing is carried out, the established farmland three-dimensional model with spectrum characteristics is a farmland soil health three-dimensional model, and the farmland topography and drainage conditions are analyzed through the farmland soil health three-dimensional model, so that optimized cultivation and water resource management advice is provided.

According to the method, when pest and disease damage is monitored, aerial hyperspectral image data of a farmland are farmland crop spectrum data, unmanned aerial vehicles acquire the farmland crop spectrum data periodically in a crop growing period, agricultural machinery acquires the farmland hyperspectral image data during fertilization or irrigation, and the pest and disease damage area distribution is obtained according to a farmland three-dimensional model with spectral characteristics, so that data support is provided for accurate pesticide application.

As a preferable scheme, when the crop growth condition is evaluated, the aerial hyperspectral image data of the farmland are farmland crop spectrum data, the unmanned aerial vehicle periodically acquires the farmland hyperspectral image data in the crop growing period, the ground hyperspectral image data are ground crop spectrum data, the agricultural machinery acquires the data when sowing, fertilizing or harvesting, and crop type identification, growth condition evaluation and yield prediction are performed according to a farmland three-dimensional model with spectrum characteristics.

According to the invention, the unmanned aerial vehicle is used for acquiring the aerial data of the farmland, the agricultural machine is used for continuously acquiring the ground data in real time, the data is acquired in the operation process of the agricultural machine, the additional investment of time and labor is not needed, the advantages of the aerial platform and the ground platform are fused in an air-ground cooperative mode, the comprehensive acquisition of the remote sensing data of the farm is realized, the high-spectrum resolution and the multi-angle acquisition are realized, the space and the spectrum resolution of the data are improved, the requirement of accurate agriculture on the data is met, and the aerial oblique photography is used for acquiring the multi-view information of the farmland to establish a high-precision farmland three-dimensional model. The invention can solve the problems of insufficient data acquisition precision and timeliness in the prior art, remarkably improves the remote sensing data acquisition precision in agricultural production, and is widely applied to the fields of crop health monitoring, pest control, soil management and the like.

Detailed Description

The following description of the embodiments of the present invention will clearly and fully describe the embodiments of the present invention, and the described embodiments are only some of the embodiments of the present invention, and not all of the embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without any inventive effort, are intended to be within the scope of the invention.

Examples:

A hyperspectral oblique photography method based on unmanned aerial vehicle and ground agricultural machinery comprises the following steps: S1, setting the synchronism of an unmanned aerial vehicle and an agricultural machine;

The unmanned aerial vehicle is provided with a positioning module, a gesture sensor, a hyperspectral camera and a tilt camera, the unmanned aerial vehicle can be a multi-rotor or fixed wing, a high-precision positioning system and a gesture control system are provided, the positioning system uses a GPS system or an RTK (Real-TIME KINEMATIC) carrier phase difference technology, the gesture sensor corresponds to the gesture control system, the gesture control system uses an inertial measurement unit IMU (Inertial measurement unit), and the unmanned aerial vehicle is provided with a communication device for transmitting collected data and receiving instructions in Real time.

The agricultural machine can be a tractor, a seeder or a harvester, has the advantages of stable work, close to the ground and convenient installation of high-performance computing equipment, is provided with a hyperspectral camera, a GPS/RTK system and an attitude sensor, ensures accurate space positioning of ground data, can continuously record the ground data in real time, and synchronizes the ground hyperspectral image data with the operation time and position information of the agricultural machine. The agricultural machinery is also provided with a storage device, a communication device and a high-performance processor, wherein the communication device is used for receiving air data transmitted by the unmanned aerial vehicle, the storage device is used for storing the air data and ground data, and the processor is used for processing the air data and the ground data.

Before the aerial data and the ground data are acquired, the unmanned aerial vehicle and the agricultural machine adopt a unified time synchronization mechanism, so that the aerial data acquired by the unmanned aerial vehicle and the ground data acquired by the agricultural machine have the same time reference;

In order to enable the aerial data collected by the unmanned aerial vehicle and the ground data collected by the agricultural machinery to have the same time standard, the time synchronization mechanism which can be adopted by the time synchronization mechanism comprises a GNSS time service mechanism, a network clock protocol mechanism, a timing signal trigger mechanism or a ground control station time unified instruction mechanism.

The GNSS timing mechanism is characterized in that GNSS modules supporting high-precision time synchronization are respectively installed on the unmanned aerial vehicle and the agricultural machine by using a global navigation satellite system GNSS (GlobalNavigationSatelliteSystem) such as a GPS (global positioning system), beidou and the like. The time signals provided by the GNSS satellites are accurate and uniform, and the time synchronization of the unmanned aerial vehicle and the agricultural machine can be ensured. And each time the GNSS modules on the unmanned aerial vehicle and the agricultural machine receive the same satellite time service signals, the unmanned aerial vehicle and the agricultural machine can start to synchronously acquire data under the time reference.

The network clock protocol mechanism is to establish a wireless network between the unmanned aerial vehicle and the agricultural machinery by adopting a network time protocol NTP or a precision time protocol PTP, wherein the unmanned aerial vehicle and the agricultural machinery are in the same network, a network time server provides a time reference, and the network time server performs time synchronization by using NTP (Network Time Protocol) or PTP (PrecisionTime Protocol). The NTP can keep the consistency of the unmanned aerial vehicle or agricultural machinery time at the millisecond level through the network time server, and the PTP has higher synchronous precision (microsecond level) and is suitable for scenes with higher requirements on the time consistency.

The timing signal triggering mechanism is to use a special timing signal generator, a receiving module is arranged on the unmanned aerial vehicle and the agricultural machinery, and the timing signal generator sends synchronous signals (such as radio pulses, wireless beacons and the like) at fixed frequency. And after receiving the signals, the unmanned aerial vehicle and the agricultural machine simultaneously start to collect data and uniformly record the collection start time, and keep synchronization on the basis. The synchronization mechanism may set a periodic trigger to keep the acquisition time points consistent. The triggering mode is simple and feasible, and is suitable for being used in environments with complex terrains and unstable network signals.

The ground control station time unified instruction mechanism is to use the ground control station as a unified control center to monitor and control the synchronism of the unmanned aerial vehicle and the agricultural machinery in real time. The ground control station sends a synchronous command to the unmanned aerial vehicle and the agricultural machinery through a wireless network, so that the unmanned aerial vehicle and the agricultural machinery start data acquisition when receiving the command. The method is suitable for controlling the scene with stable conditions, and the synchronization mechanism can be applied together with the GNSS time service mechanism and the network clock protocol mechanism, so that the synchronism of a plurality of devices is ensured.

Of course, a time stamp alignment and post-processing synchronization mechanism can also be used, and alignment is performed according to the time stamp records of each device in the data post-processing stage. The above-mentioned several synchronization mechanisms are generally real-time synchronization mechanisms, and the time stamp alignment and post-processing mechanisms are applicable to the situation that real-time is not required, and if real-time synchronization is difficult to realize, the time of ground and air data can be matched through time stamp alignment during post-processing of the data. Although this approach is not synchronized in real time, it is still effective for most agricultural data acquisition scenarios.

Of course, the synchronous signal line or frequency synchronization can be used, if the unmanned aerial vehicle and the agricultural machine are close, the synchronous signal can be transmitted by using a physical connection (such as an optical fiber or a cable), so that the equipment directly shares a unified clock signal source. The method is suitable for scenes where the unmanned aerial vehicle is connected with the agricultural machinery or the distance is short, and the time references of the unmanned aerial vehicle and the agricultural machinery are ensured to be completely consistent.

By adopting the synchronization mechanism, the unmanned aerial vehicle and the agricultural machinery can share the same time reference in the data acquisition process, so that high-precision synchronization of aerial and ground data is realized, and reliable data support is provided for multi-source data fusion and farmland three-dimensional modeling.

S2, acquiring aerial data by the unmanned aerial vehicle;

The unmanned aerial vehicle flies according to set conditions to obtain aerial data, wherein the aerial data comprise aerial hyperspectral image data of farmlands, aerial inclined image data of farmlands, unmanned aerial vehicle position data and unmanned aerial vehicle attitude data.

The set conditions include flight path, flight altitude, flight speed, course interval, hyperspectral camera inclination angle and oblique photographic camera inclination angle.

The hyperspectral camera is used for acquiring hyperspectral image data of farmland, and can be hyperspectral image data of single crops, hyperspectral image data of single soil and hyperspectral image data of both crops and soil;

The oblique photography camera is used for acquiring oblique photography image data of farmlands;

and the positioning module and the gesture sensor generate corresponding position information and gesture information in the unmanned aerial vehicle flight process.

S3, acquiring ground data by the agricultural machinery;

continuously acquiring ground data by the agricultural machinery in the farmland operation process, wherein the ground data comprises ground hyperspectral image data, agricultural machinery position data and agricultural machinery posture data;

the method comprises the steps of obtaining agricultural machinery position data through a positioning module of the agricultural machinery, obtaining agricultural machinery posture data through a posture sensor, obtaining ground hyperspectral image data through a hyperspectral camera, wherein the ground hyperspectral image data can be soil hyperspectral image data, crop hyperspectral image data, and hyperspectral image data of soil and crops.

S4, data preprocessing;

Preprocessing the aerial hyperspectral image data;

Performing radiation correction and atmosphere correction on the aerial hyperspectral image data;

carrying out geometric correction on the aerial hyperspectral image data subjected to radiation correction and atmosphere correction according to the unmanned aerial vehicle position data and the unmanned aerial vehicle posture data;

Preprocessing the aerial inclined image data;

Geometrically correcting the air inclined image data of the farmland according to the unmanned aerial vehicle position data and the unmanned aerial vehicle attitude data;

preprocessing ground hyperspectral image data;

performing radiation correction and atmosphere correction on the ground hyperspectral image data;

the ground hyperspectral image data after the radiation correction and the atmosphere correction are geometrically corrected according to the agricultural machinery position data and the agricultural machinery posture data;

s5, carrying out data fusion on the ground hyperspectral image data and the aerial hyperspectral image data;

The method comprises the steps of firstly, carrying out space matching on the ground hyperspectral image data after pretreatment and the air hyperspectral image data after pretreatment, correcting the space deviation of the air hyperspectral image data of the unmanned aerial vehicle, wherein the ground hyperspectral image data after pretreatment and the air hyperspectral image data have consistent spectral response, and the space matching ensures the accurate alignment of the ground hyperspectral image data and the air hyperspectral image data in space, so that a foundation is provided for subsequent data fusion.

The specific steps of space matching are as follows:

unified coordinate system and geometric correction:

The ground hyperspectral image data and the aerial hyperspectral image data are converted into the same geographic coordinate system (such as WGS84 coordinate system) by using the GNSS and IMU data so as to align the geographic positions of the images. The GNSS provides geographic position and the IMU provides attitude data (e.g., tilt angle, rotation angle, etc.), which, in combination, can accurately position the image into geographic coordinates.

Then, geometric correction is carried out on the image data in the same geographic coordinate system, so that the influence of inclination, rotation and the like generated in the flight process of the unmanned aerial vehicle can be corrected, the geographic position of the image is more approximate to the actual ground position, the spatial geometric deformation of the inclined image is ensured to be corrected, and the image is accurately corresponding to the ground coordinate.

Digital Elevation Model (DEM) correction:

For areas of irregular terrain or large differences in elevation, correction may be made using digital elevation model DEM (Digital Elevation Model) or digital surface model DSM (DigitalSurface Model),

The unmanned aerial vehicle shooting area has the topographic relief or large height difference, and can be further corrected by combining a Digital Elevation Model (DEM). The DEM can provide ground height information of each position, the inclined images are subjected to terrain registration by using the DEM data, the images are mapped onto real terrains, spatial deviation caused by terrain fluctuation is corrected, and accurate alignment of the images and the actual terrains is ensured.

Ground Control Point (GCP) calibration:

Ground Control Points (GCPs) are set in the farmland and their exact geographic coordinates are recorded. The control points in the unmanned aerial vehicle image are in one-to-one correspondence with the ground control points, so that the space deviation can be further reduced.

And registering the images by using the known coordinate points to ensure the spatial precision of the aerial images. GCP is typically selected as a distinct ground feature (e.g., road intersection, marker, etc.) for easy identification in images.

Feature point detection and matching:

The key geographic feature points are respectively extracted from the ground hyperspectral image data and the air hyperspectral image data through a feature extraction algorithm, and the key address feature points generally comprise obvious landmarks, road intersection points, ridges and the like, and can be clearly identified in the ground and air images.

The adopted feature extraction algorithm can be an algorithm SIFT (Scale-INVARIANT FEATURE TRANSFORM) for detecting local features, a feature algorithm SURF (SpeedUp Features) with robustness of an acceleration edition or an algorithm ORB (Oriented FAST and Rotated BRIEF) for extracting and describing feature points rapidly;

And matching the key geographic feature points, and establishing spatial correlation of ground and aerial images through matched point pairs to provide a reference for the next spatial registration.

Image resampling and spatial registration:

The ground image is resampled to a spatial scale and resolution consistent with the aerial image using a geometric transformation (e.g., affine transformation or perspective transformation) based on the matching key geographic feature points, such that the ground image is geographically aligned with the aerial image. The geometric transformation can eliminate geometric deformation of the image due to different visual angles and heights, so that the image has the same spatial scale as the ground image.

Before spatial registration, stereo matching or point cloud generation technology (such as Structure-from-Motion, sfM) is required to be used for the aerial image data obtained by oblique photography, multi-view image fusion is carried out, global optimization reconstruction is carried out to obtain three-dimensional point cloud after multi-view image fusion, and accurate alignment of ground images and aerial images on spatial coordinates is ensured.

The aligned images are resampled to be adjusted to be consistent in resolution, accurate correspondence on the image space is ensured, and a foundation is laid for spectrum fusion.

Verification and error correction:

By verifying the alignment accuracy of a typical area (such as a landmark point) in an image, namely calculating the offset of the landmark point in the image, the ground image and the aerial image are ensured to be matched accurately in space. If there is a deviation, the feature points and registration parameters (e.g., transformation matrix, resampling resolution) can be further adjusted. And calculating the precision after registration by using an error analysis method, and ensuring that the space matching meets the requirements.

Through the steps, accurate spatial matching of ground and aerial hyperspectral images can be realized, and a stable foundation is provided for further fusion of spectrum data and hyperspectral feature construction of a farmland three-dimensional model.

And after space matching, carrying out spectrum fusion on the ground hyperspectral image data and the aerial hyperspectral image data to obtain fused hyperspectral image data.

The specific steps of spectrum fusion are as follows:

Spectral normalization:

Because the spectra of the ground and aerial images may vary depending on the acquisition height and sensor characteristics, the images are first spectrally normalized to match the spectral intensity to the color response, and radiation correction and atmospheric correction methods (e.g., FLAASH or QUAC) may be used to normalize to reduce the spectral difference between the ground and aerial data.

And an atmospheric correction algorithm (e.g., FLAASH or QUAC) to eliminate spectral deviations due to atmospheric scattering and absorption, thereby ensuring that the ground and air data are spectrally consistent.

Spectral band matching and resampling:

the ground and aerial image data are aligned to a uniform spectral band. If the data spectrum wave band distribution of the two groups of images is different, the inconsistent wave bands can be interpolated or resampled, so that the two images correspond to each other on the same wave band.

And processing the data of different wave bands by adopting an interpolation method (such as linear interpolation and spline interpolation) to ensure that all wave band information can be matched to the same spectrum scale.

Spectral weighted fusion:

And aiming at the overlapping area after space matching, adopting a weighted fusion strategy to carry out weighted synthesis on the spectral details of the ground image and the whole information of the aerial image. So that the spectral information near the ground is more prominent in detail, and the aerial image is more macroscopically specific.

Different weights can be set according to the spatial resolution and visual difference of the images, and the ground images generally provide high-resolution details, so the weights are slightly high, the aerial images provide wide-area information, and the weights are slightly low. The fused spectrum information has the characteristics of both global visual angle and detail information.

Multiscale fusion:

and adopting a multi-scale fusion method (such as wavelet transformation or pyramid fusion) to fuse different scale information of the ground image and the aerial image, reserving integral information of the aerial image at a low-frequency level and reserving detail information of the ground image at a high-frequency level. The wavelet transformation can decompose different scale information of the image, integrate high-frequency detail information of the ground image with low-frequency structure information of the aerial image, so that the details of the ground image are highlighted in a high-frequency part, and the integral structure of the aerial image is reserved in a low-frequency part.

And reconstructing after multi-scale fusion to generate complete hyperspectral image data containing high-frequency and low-frequency information, and retaining the details and integral information of the two.

Spectral smoothing and detail optimization:

And smoothing the fused spectrum data to reduce abrupt change of the spectrum data at the boundary, so that transition between the image data of different sources is natural.

The smoothing process uses a spectral smoothing algorithm (such as Gaussian smoothing or moving average) to process the spectral data, so that the spectral continuity of the image is enhanced, the naturalness of the spectral information is improved, and the spectral information is more continuous and real.

Outputting fused hyperspectral image data:

and storing the fused hyperspectral image data into a multiband image file (such as GeoTIFF format or HDF format or ENVI format) or a hyperspectral image data format, so as to ensure that each pixel contains complete spectral information.

The output fusion data can be directly used for further analysis (such as three-dimensional modeling or vegetation index calculation) and can also be used for visual display of hyperspectral images.

Through the spectrum fusion steps, fused hyperspectral image data with continuous space and rich spectrum can be generated. The image data not only maintains the detailed information of the ground image, but also combines the whole visual angle of the aerial image, and is suitable for the fine application of agricultural monitoring, crop growth analysis and the like.

The ground hyperspectral image data are fine and close, the defects of the aerial hyperspectral image data can be corrected and supplemented, and the spectral resolution and the spatial precision of the whole data are improved;

s6, constructing a farmland three-dimensional model with spectral characteristics;

The three-dimensional model of the farmland is constructed by using the preprocessed air inclined image data, which are collected multi-view images of the unmanned aerial vehicle for obliquely shooting the farmland from different angles and different heights, and can be generally realized by a multi-view three-dimensional reconstruction technology, so that the coverage range and the reconstruction precision are ensured. And radiation correction, geometric correction and atmospheric correction are performed on the aerial inclined image data during preprocessing, so that illumination difference, perspective deformation and atmospheric interference are removed, and the image data is more accurate.

The method specifically comprises the following steps:

Feature point detection and matching:

Feature extraction algorithms (e.g., SIFT, SURF, or ORB) are used to detect feature points in each image. These feature points should be consistent at different viewing angles to facilitate matching.

And matching the characteristic points in the multi-view images to find out the corresponding points in different images. The matching result forms an initial spatial geometrical relationship and provides a basis for three-dimensional reconstruction.

Structural-from-Motion (SfM):

A structural reconstruction algorithm (such as Structure-from-Motion) is used to generate a camera pose and an initial sparse point cloud based on the feature point matching results. The SfM algorithm restores the pose and the space position of the camera through the feature point matching result of the multiple images, generates a sparse point cloud, and constructs the sparse three-dimensional point cloud of the scene through resolving camera external parameters (pose and position). The sparse point cloud generated at the stage reflects the basic spatial layout of farmlands, roughly outlines a scene structure, is a basic framework of a model, and has lower resolution and less details.

Generating a dense point cloud:

On the basis of sparse point clouds, dense matching algorithms (such as Multi-View Stereo, MVS) are used to generate high-density point clouds. MVS restores more points to three-dimensional space based on disparities between multi-view images. The generated dense point cloud has higher resolution and rich details, and can accurately reflect fluctuation of farmland topography and surface morphology of crops.

Gridding and surface reconstruction:

And performing triangular meshing (such as Delaunay triangulation) on the dense point cloud data to generate a farmland three-dimensional grid model, wherein the meshing step is used for connecting the point cloud into a surface and providing a surface structure for the three-dimensional model.

And after gridding, carrying out surface smoothing and optimizing treatment to eliminate noise points or irregular surfaces, so that the surface of the model is smoother and more real.

Mapping the hyperspectral image data to a farmland three-dimensional model to obtain a farmland three-dimensional model with spectral characteristics;

The operation steps are as follows:

three-dimensional model and spatial calibration of fused hyperspectral image data:

The three-dimensional model and the coordinate system of the fused hyperspectral image data are aligned, and consistent geographic coordinates can be provided for the image and the three-dimensional model through GNSS and IMU data, so that the fused hyperspectral image data are matched with the spatial position of the farmland three-dimensional model.

If the three-dimensional model uses multi-view reconstruction (such as those generated by SfM and MVS), it is ensured that its position, scale and orientation are consistent with the hyperspectral image data. And the inclination angle and the terrain height difference are corrected in an auxiliary mode through a Digital Elevation Model (DEM), so that the image is ensured to be matched with the ground surface position of the three-dimensional model in a space accurately.

Feature point matching and image projection:

Common geographic feature points (such as landmarks, road intersections, ridges, roads and the like) are found in the farmland three-dimensional model and the fused hyperspectral image data, and feature matching algorithms (such as SIFT, SURF and the like) are used for registering the feature points so as to ensure that the fused hyperspectral image data and the three-dimensional model are accurately aligned.

The fused hyperspectral image data is projected onto the surface of the three-dimensional model using a projective transformation algorithm (e.g., orthographic projection or perspective projection) such that each three-dimensional grid or point cloud point corresponds to a pixel in the fused hyperspectral image data.

And (3) layering and mapping the fused hyperspectral image data to the surface of the farmland three-dimensional model:

And mapping each band information fused with the hyperspectral image data as an independent texture layer onto the surface texture of the farmland three-dimensional model layer by layer. In general, each band fused with hyperspectral image data represents different spectral features, and the band data is mapped as texture information to each surface grid of the farmland three-dimensional model in a layered manner, so that the spectral information of each band can be directly superimposed on the texture of the three-dimensional model.

And directly mapping the pixel values fused with the hyperspectral image data to the corresponding surface positions of the three-dimensional model by using an image projection or ray projection technology, and generating a texture image layer containing spectral information, thereby retaining rich spectral characteristics.

Spectral data interpolation and weighted fusion:

For the part which is not directly aligned to the farmland three-dimensional model grid in the fused hyperspectral image data, an interpolation algorithm (such as bilinear interpolation or spline interpolation) can be used for filling the unaligned part, so that each grid point is ensured to have complete spectral data.

And a weighted fusion strategy can be used in the overlapped area to fuse the hyperspectral data of a plurality of visual angles to the surface of the model, so that the accuracy and the continuity of the spectrum information of the model are enhanced.

Multiband texture generation:

And layering the data of each wave band on the texture of the farmland three-dimensional model to generate a texture image containing multiple wave bands. Thus, a three-dimensional model with hyperspectral characteristics can be generated, and analysis and observation under different spectral bands are facilitated.

The surface unit (such as triangle patch or point) of each farmland three-dimensional model contains multi-band spectrum data, and can be used for subsequent vegetation health analysis, crop condition evaluation and other applications.

Model optimization and spectral feature extraction:

and optimizing the farmland three-dimensional model with the spectrum information to ensure that the hyperspectral data has smooth transition on the surface of the farmland three-dimensional model, and avoid abrupt change or discontinuity.

And further calculating spectral indexes (such as NDVI, NDRE and the like) and mapping the spectral indexes onto a farmland three-dimensional model, so that the farmland three-dimensional model presents spectral characteristic information of farmland, and the growth condition and health state of crops can be conveniently identified.

Outputting a three-dimensional model with spectral features:

the finally generated three-dimensional model with spectral features is exported into a format (such as PLY, OBJ, etc.) supporting multi-band data for subsequent further investigation in three-dimensional visualization software or analysis platforms.

The output model can be directly used for agricultural monitoring and accurate management, and different crop areas can be deeply analyzed by observing spectral characteristic information of the three-dimensional model.

Through the hyperspectral data mapping, the farmland three-dimensional model has accurate spectral characteristics, can support multi-level farmland information analysis, and is convenient for the fine evaluation and management of crop conditions.

After the farmland three-dimensional model with spectral characteristics is obtained, the method can be applied to crop growth assessment, pest and disease damage monitoring and soil health condition analysis.

Crop species identification, growth assessment and yield prediction can be performed according to the three-dimensional model of the farmland with spectral features and the hyperspectral features of crops.

The practical implementation is as follows:

The unmanned aerial vehicle is used for regularly acquiring and collecting aerial data of a large-area farmland in a crop growth period, the agricultural machine is used for acquiring hyperspectral image data of crops in sowing, fertilizing or harvesting, and crop type identification, growth assessment and yield prediction are carried out according to a farmland three-dimensional model with spectral characteristics.

And identifying the occurrence area of the plant diseases and insect pests by utilizing spectrum anomaly detection according to the farmland three-dimensional model with the spectrum characteristics, and guiding accurate control.

By utilizing a hyperspectral imaging technology, spectral anomalies caused by diseases and insect pests are found by analyzing spectral reflection characteristics of crops, and a farmland three-dimensional model with spectral characteristics is combined to position a disease and insect pest area, so that farmers are guided to carry out precise control, pesticide use is reduced, and crop health is improved.

The practical implementation is as follows:

the unmanned aerial vehicle regularly carries out flight monitoring in the crop growing period, acquires hyperspectral images of crops, and detects early disease and pest characteristics.

When the agricultural machinery is used for fertilizing or irrigating, the plant diseases and insect pests in the heavy-spot area are secondarily verified by collecting near-ground hyperspectral data of crops.

And obtaining the distribution of the pest and disease damage areas according to a farmland three-dimensional model with spectral characteristics, and providing data support for accurate pesticide application.

And monitoring the conditions of organic matters, water content and nutrients in farmland soil according to the farmland three-dimensional model with spectral characteristics, and carrying out soil health conditions.

The practical implementation is as follows:

The unmanned aerial vehicle obtains soil surface layer spectrum data of a large-area farmland to generate a soil organic matter and water content distribution diagram, the agricultural machine obtains ground soil spectrum data when fertilizing or sowing, verifies key areas in the unmanned aerial vehicle data, a farmland three-dimensional model with spectrum characteristics, which is established by the soil surface layer spectrum data and the ground soil spectrum data, is a farmland soil health three-dimensional model, analyzes farmland topography and drainage conditions through the farmland soil health three-dimensional model, and provides optimized cultivation and water resource management advice.

The unmanned aerial vehicle is combined with the ground agricultural machinery to realize the cooperative data acquisition of the air and the ground, so that the space and the spectral resolution of the data are improved, the three-dimensional information of crops and soil is acquired, the acquisition of high-precision and multi-angle data is realized, the agricultural machinery can acquire the hyperspectral data in daily operation without additional investment time and cost, the data acquisition efficiency is improved, the monitoring precision of the crops and the soil is improved through the fusion of the air and the ground data, the method has obvious advantages in the aspects of crop growth monitoring, pest and disease identification, soil health evaluation and the like, the method can be suitable for large-area farmland monitoring, can also be used in small-area test fields, has good adaptability and flexibility, has wide application prospects in the fields of precise agriculture, environment monitoring and the like, and can remarkably improve the agricultural production efficiency and the resource utilization rate.

In the description of the present specification, reference to the terms "one embodiment," "example," "specific example," and the like, means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

The foregoing is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art, who is within the scope of the present invention, should make equivalent substitutions or modifications according to the technical scheme of the present invention and the inventive concept thereof, and should be covered by the scope of the present invention.

Claims (10)

1. S1, setting the synchronicity of the unmanned aerial vehicle and the agricultural machinery, and enabling the unmanned aerial vehicle and the agricultural machinery to adopt a uniform time synchronization mechanism so that the aerial data collected by the unmanned aerial vehicle and the ground data collected by the agricultural machinery have the same time standard;

s2, acquiring aerial data by the unmanned aerial vehicle;

the aerial data comprise aerial hyperspectral image data of farmlands, aerial inclined image data of farmlands, unmanned aerial vehicle position data and unmanned aerial vehicle attitude data;

S3, acquiring ground data by the agricultural machinery;

the ground data comprise ground hyperspectral image data, agricultural machinery position data and agricultural machinery posture data;

S4, data preprocessing;

Preprocessing the aerial hyperspectral image data;

Performing radiation correction and atmosphere correction on the aerial hyperspectral image data;

carrying out geometric correction on the aerial hyperspectral image data subjected to radiation correction and atmosphere correction according to the unmanned aerial vehicle position data and the unmanned aerial vehicle posture data;

Preprocessing the aerial inclined image data;

Geometrically correcting the air inclined image data of the farmland according to the unmanned aerial vehicle position data and the unmanned aerial vehicle attitude data;

preprocessing ground hyperspectral image data;

performing radiation correction and atmosphere correction on the ground hyperspectral image data;

the ground hyperspectral image data after the radiation correction and the atmosphere correction are geometrically corrected according to the agricultural machinery position data and the agricultural machinery posture data;

s5, carrying out data fusion on the ground hyperspectral image data and the aerial hyperspectral image data;

s6, constructing a farmland three-dimensional model with spectral characteristics;

constructing a farmland three-dimensional model by using the preprocessed air inclined image data;

and mapping the hyperspectral image data to the farmland three-dimensional model to obtain the farmland three-dimensional model with spectral characteristics.

2. The hyperspectral oblique photography method based on the unmanned aerial vehicle and the ground agricultural machinery as claimed in claim 1, wherein the unmanned aerial vehicle is provided with a positioning module, an attitude sensor, a hyperspectral camera and an oblique photography camera, and flies according to set conditions, wherein the set conditions comprise a flight path, a flight altitude, a flight speed, a route interval, a hyperspectral camera oblique angle and an oblique photography camera oblique angle.

3. The hyperspectral oblique photography method based on the unmanned aerial vehicle and the ground agricultural machinery as claimed in claim 2, wherein the agricultural machinery is provided with a positioning module, an attitude sensor and a hyperspectral camera.

4. The hyperspectral oblique photography method based on the unmanned aerial vehicle and the ground agricultural machinery according to claim 3 is characterized in that the specific step of data fusion is that the preprocessed ground hyperspectral image data and the preprocessed air hyperspectral image data are subjected to space matching;

And after space matching, carrying out spectrum fusion on the ground hyperspectral image data and the aerial hyperspectral image data to obtain fused hyperspectral image data.

5. The hyperspectral oblique photography method based on the unmanned aerial vehicle and the ground agricultural machinery as claimed in claim 4, wherein the specific steps of the space matching are that a coordinate system and geometric correction are unified:

Converting the ground hyperspectral image data and the aerial hyperspectral image data into the same geographic coordinate system by using the GNSS and IMU data of the global navigation satellite system to align the geographic positions of the images;

digital elevation model DEM correction:

For areas with irregular terrains or large height differences, correcting by using a Digital Elevation Model (DEM);

ground control point GCP calibration:

setting a ground control point GCP in a farmland, and recording accurate geographic coordinates of the GCP; registering the images by using the known coordinate points to ensure the space precision of the aerial images;

Feature point detection and matching:

Extracting key geographic feature points from the ground hyperspectral image data and the air hyperspectral image data respectively through a feature extraction algorithm, wherein the key address feature points generally comprise obvious landmarks, road intersection points, ridges and the like, and the key geographic feature points can be clearly identified in the ground and air images;

matching the key geographic feature points;

image resampling and spatial registration:

and resampling the ground image to a spatial scale and resolution consistent with the aerial image using geometric transformations based on the matched key geographic feature points, such that the ground image is geographically aligned with the aerial image.

6. The hyperspectral oblique photography method based on the unmanned aerial vehicle and the ground agricultural machinery as claimed in claim 5, wherein the specific steps of spectrum fusion are,

Spectral normalization:

Spectral band matching and resampling:

If the data spectrum wave bands of the two groups of images are different in distribution, interpolation or resampling can be carried out on inconsistent wave bands so that the two data spectrum wave bands correspond to each other on the same wave band;

Spectral weighted fusion:

aiming at the overlapping area after space matching, a weighted fusion strategy is adopted to carry out weighted synthesis on the spectrum details of the ground image and the whole information of the aerial image, the ground hyperspectral image data weight is large, and the aerial hyperspectral image data weight is low;

Multiscale fusion:

The method comprises the steps of adopting a multi-scale fusion method to fuse different scale information of ground hyperspectral image data and aerial hyperspectral image data, reserving integral information of aerial images at a low-frequency level, and reserving detail information of ground images at a high-frequency level;

reconstructing after multi-scale fusion to generate complete hyperspectral image data containing high-frequency and low-frequency information, and reserving details and integral information of the two;

Spectral smoothing and detail optimization:

Smoothing the fused spectrum data to reduce abrupt change of the spectrum data at the boundary, so that transition between the image data of different sources is natural;

outputting fused hyperspectral image data:

and storing the fused hyperspectral image data as a multiband image file.

7. The hyperspectral oblique photography method based on the unmanned aerial vehicle and the ground agricultural machinery according to claim 6, wherein the construction step of the farmland three-dimensional model is characterized in that the characteristic points are detected and matched:

Matching the characteristic points in the multi-view images to find corresponding points in different images;

And (3) structural reconstruction:

Generating a camera pose and an initial sparse point cloud based on the feature point matching result by using a structure reconstruction algorithm;

generating a dense point cloud:

Generating a high-density point cloud by using a dense matching algorithm on the basis of the sparse point cloud;

Gridding and surface reconstruction:

Performing triangular gridding on the dense point cloud data to generate a farmland three-dimensional grid model, wherein the gridding step connects the point cloud into a surface to provide a surface structure for the three-dimensional model;

After gridding, carrying out surface smoothing and optimization treatment to eliminate noise points or irregular surfaces, so that the surface of the model is smoother and more real;

The operation steps of mapping the hyperspectral image data to the farmland three-dimensional model are as follows:

three-dimensional model and spatial calibration of fused hyperspectral image data:

aligning the three-dimensional model with a coordinate system of the fused hyperspectral image data to enable the fused hyperspectral image data to be matched with the space position of the farmland three-dimensional model;

feature point matching and image projection:

searching common geographic feature points in the farmland three-dimensional model and the fused hyperspectral image data, and registering the feature points by using a feature matching algorithm;

Projecting the fused hyperspectral image data onto the surface of the three-dimensional model by using a projection transformation algorithm, so that each three-dimensional grid or point cloud point can correspond to a pixel in the fused hyperspectral image data;

and (3) layering and mapping the fused hyperspectral image data to the surface of the farmland three-dimensional model:

Each band information fused with hyperspectral image data is used as an independent texture layer and mapped onto the surface texture of the farmland three-dimensional model layer by layer;

using image projection or ray projection technology to directly map the pixel value of the fused hyperspectral image data to the corresponding surface position of the three-dimensional model, and generating a texture image layer containing spectral information;

Spectral data interpolation and weighted fusion:

For the part which is not directly aligned to the farmland three-dimensional model grid in the fused hyperspectral image data, filling the unaligned part by using an interpolation algorithm, and ensuring that each grid point has complete spectral data;

using a weighted fusion strategy in the overlapping area to fuse hyperspectral data of a plurality of view angles to the surface of the model;

multiband texture generation:

Layering the data of each wave band on the texture of the farmland three-dimensional model to generate a texture image containing multiple wave bands;

Model optimization and spectral feature extraction:

Optimizing the farmland three-dimensional model with the spectrum information to ensure that the hyperspectral data has smooth transition on the surface of the farmland three-dimensional model, and avoiding abrupt change or discontinuity;

calculating the spectrum index and mapping the spectrum index to the farmland three-dimensional model, so that the farmland three-dimensional model presents spectrum characteristic information of farmland;

Outputting a three-dimensional model with spectral features:

the finally generated three-dimensional model with spectral features is exported into a format supporting multiband data.

8. The hyperspectral oblique photography method based on the unmanned aerial vehicle and the ground agricultural machinery, which is characterized in that the three-dimensional model of the farmland with spectral characteristics is applied to crop growth assessment, pest and disease damage monitoring and soil health analysis.

9. The hyperspectral oblique photography method based on the unmanned aerial vehicle and the ground agricultural machinery, which is disclosed by claim 8, is characterized in that when the soil health condition analysis is carried out, the aerial hyperspectral image data of the farmland are farmland soil surface layer spectral data, the ground hyperspectral image data are ground soil spectral data, the agricultural machinery is obtained when fertilization or sowing is carried out, the established farmland three-dimensional model with spectral characteristics is a farmland soil health three-dimensional model, and the farmland topography and drainage conditions are analyzed through the farmland soil health three-dimensional model, so that optimized cultivation and water resource management suggestions are provided.

10. The hyperspectral oblique photography method based on the unmanned aerial vehicle and the ground agricultural machinery, which is disclosed in claim 8, is characterized in that when the plant and insect pest monitoring is carried out, the aerial hyperspectral image data of the farmland are farmland crop spectrum data, the unmanned aerial vehicle is obtained periodically in the growing period of crops, the ground hyperspectral image data are ground crop spectrum data, the agricultural machinery is obtained during fertilization or irrigation, the plant and insect pest area distribution is obtained according to a farmland three-dimensional model with spectral characteristics, and data support is provided for accurate pesticide application;

When the crop growth condition is evaluated, the aerial hyperspectral image data of the farmland are farmland crop spectrum data, the unmanned aerial vehicle periodically acquires the farmland hyperspectral image data in the crop growing period, the ground hyperspectral image data are ground crop spectrum data, the agricultural machinery acquires the data when sowing, fertilizing or harvesting, and crop type identification, growth condition evaluation and yield prediction are performed according to a farmland three-dimensional model with spectrum characteristics.