CN116989772B - An air-ground multi-modal multi-agent collaborative positioning and mapping method - Google Patents

Abstract

The invention proposes a multi-modal multi-agent collaborative positioning and mapping method in the air and ground, which includes obtaining measurement data of the intelligent agents. The intelligent agents include drones and unmanned vehicles, and the unmanned vehicles are equipped with visual markers; The measurement data of people and vehicles are used for local mapping from a local perspective, and the measurement data of drones are used for local mapping from a global perspective; visual markers are detected through drones, and when the detection is successful, the relationship between drones and unmanned vehicles is obtained. The relative pose between the objects is used to fuse and optimize the air-ground perspective map using the conversion relationship of the relative pose; based on loop closure, it is continuously detected whether the agent passes through the overlapping area. When the overlapping area is detected, two images of the agent are established through matching key frames. Map correlation, trajectory calibration and similar perspective map fusion and optimization are performed; based on the map fused and optimized from the air and ground perspective and the map fused and optimized from the similar perspective, a map with consistent pose trajectory and global consistency is obtained.

Description

An air-ground multi-modal multi-agent collaborative positioning and mapping method

Technical field

The invention belongs to the field of positioning and mapping of unmanned systems.

Background technique

With the continuous technological breakthroughs in the field of intelligent robots, intelligent robots are increasingly used to solve complex problems in modern society, and autonomous navigation capabilities are considered to be the basis for intelligent robots to achieve autonomous tasks. In order to realize robot autonomous planning and navigation, how to make intelligent robots coordinate positioning and mapping (SLAM) has become a hot research topic at present, especially in the fields of military, disaster environment search and rescue, etc. It has huge potential.

At present, mutual pose correction and information perception among multiple robots have become a difficulty. During the SLAM process, the robot cannot obtain information about the entire environment in advance, and the construction of the map mainly relies on the robot's exploration. Traditional single robots have many shortcomings in performing SLAM in large-scale scenarios, such as small sensor range, limited observation angle, high computational complexity, and weak storage capacity. Relying only on local sensor information, the robot's positioning error will become larger and larger, eventually leading to mapping and positioning deviations. For some special tasks, such as military, disaster search and rescue, etc., the task needs to be completed in the shortest possible time. For the above problems, through the communication and mutual coordination of multiple robots scattered in the environment, the entire system can obtain stronger environmental detection capabilities and accurate positioning capabilities.

Contents of the invention

The present invention aims to solve one of the technical problems in the related art, at least to a certain extent.

To this end, the purpose of the present invention is to propose an open-ground multi-modal multi-agent collaborative positioning and mapping method for constructing a globally consistent environmental map model.

In order to achieve the above purpose, the first embodiment of the present invention proposes an air-ground multi-modal multi-agent collaborative positioning and mapping method, including:

Obtaining measurement data of an intelligent agent, the intelligent agent includes a drone and an unmanned vehicle, and the unmanned vehicle is provided with a visual marker;

Local mapping from a local perspective is performed using the measurement data of the unmanned vehicle, and local mapping from a global perspective is performed using the measurement data of the UAV;

The visual marker is detected by the drone, and when the detection is successful, the relative pose between the drone and the unmanned vehicle is obtained, and the conversion relationship between the relative pose is used to perform an air-ground perspective. Map fusion and optimization;

Continuously detect whether the agent passes through the overlapping area based on loop closure. When the overlapping area is detected, the association between the two maps of the agent is established through matching key frames, and trajectory calibration and similar perspective map fusion and optimization are performed;

According to the map after the fusion and optimization of the air and ground perspectives and the map after the fusion and optimization of the similar perspective, a map with consistent pose trajectory and global consistency is obtained.

In addition, an air-ground multi-modal multi-agent collaborative positioning and mapping method according to the above embodiments of the present invention may also have the following additional technical features:

Further, in one embodiment of the present invention, obtaining the measurement data of the intelligent agent includes:

Preprocess the measurement data, including visual detection and optical flow tracking, and inertial measurement unit IMU pre-integration; where,

The visual detection and optical flow tracking include: applying the least squares method to obtain the velocity vector of the optical flow, as follows:

,

in, ,/> Indicates that the brightness of the pixels in the image is/> ,/> Image gradient in direction, /> Shown in/> Time gradient in direction,/> ,/> is the optical flow edge/> ,/> axis velocity vector;

The inertial measurement unit IMU pre-integration includes:

,

in, is the IMU coordinate system,/> The coordinate system where the IMU is located when initializing the origin is the world coordinate system, /> and/> are the acceleration and angular velocity measured by the IMU,/> for/> Time rotation from IMU coordinate system to world coordinate system,/> It is the right multiplication of the quaternion;

General Frame to/> By integrating all the IMU data between frames, we can get the Frame position/> , speed/> and rotate/> ,/> As an initial value for visual estimation, the rotation is in quaternion form.

Further, in one embodiment of the present invention, the local perspective mapping is performed using the measurement data of the unmanned vehicle, and the global perspective local mapping is performed using the measurement data of the unmanned aerial vehicle, including:

Use SFM to solve the poses of all frames and the three-dimensional positions of all landmark points in the sliding window, and align them with the IMU pre-integrated values to obtain the angular velocity offset, gravity direction, scale factor and velocity corresponding to each frame;

Use the sliding window method to optimize the state variables within the window, in Optimized state vector in time window/> As follows,

,

in, and/> is the rotation and translation part of the camera pose,/> is the speed of the camera in the world coordinate system,/> and/> are the acceleration bias and angular velocity bias of the IMU respectively; the optimization objective function of the system's state quantity is as follows:

,

in, is the maximum estimated posterior value,/> is the sliding window initial value residual,/> is the IMU observation residual,/> is the camera observation residual.

Further, in one embodiment of the present invention, continuously detecting whether the agent passes through an overlapping area based on loop closure includes:

All visual features are clustered, one type of feature is a word, and all features are a dictionary;

Describe an image with a single vector:

Calculate the similarity between two images A and B :

,

in, is a vector describing image A, /> is a vector describing image B, /> is the similarity between two images A and B, /> Represents a vector describing image A /> of/> components,/> represents a vector describing image B of/> weight;

If the similarity exceeds the threshold, a loop is considered to have occurred.

Further, in one embodiment of the present invention, clustering all visual features includes:

At the root node, use the K-means algorithm to cluster all samples into Class, get the first layer;

For each node in the first layer, the samples belonging to the node are clustered into Class, the next layer is obtained; and so on, finally the leaf layer is obtained, and the leaf layer is the word.

Further, in one embodiment of the present invention, describing an image with a single vector includes:

definition for word/> The number of features included,/> is the number of features contained in all words,/> for words The number of times it appears in image A,/> for word/> The total number of occurrences in all images, then

word weights in image A/> for:/> ,

via bag-of-words, using a single vector Describe an image A:

,

in, is the word that image A has in the dictionary,/> yes The corresponding weight,/> is a vector describing image A.

In order to achieve the above purpose, the second embodiment of the present invention proposes an air-ground multi-modal multi-agent collaborative positioning and mapping device, which includes the following modules:

An acquisition module, used to acquire measurement data of an intelligent agent. The intelligent agent includes a drone and an unmanned vehicle, and the unmanned vehicle is provided with a visual marker;

A mapping module, configured to perform local mapping from a local perspective using the measurement data of the unmanned vehicle, and perform local mapping from a global perspective using the measurement data of the drone;

An air-ground perspective fusion module is used to detect the visual marker through the drone, and when the detection is successful, obtain the relative pose between the drone and the unmanned vehicle, using the relative pose Perform air-ground perspective map fusion and optimization based on the posture conversion relationship;

A similar perspective fusion module is used to continuously detect whether the agent passes through an overlapping area based on loop closure. When the overlapping area is detected, the association between the two maps of the agent is established through matching key frames, and trajectory calibration and Similar perspective map fusion and optimization;

An output module is used to obtain a map with pose trajectory and global consistency based on the map after fusion and optimization of the air-ground perspective and the map after fusion and optimization of the similar perspective.

In order to achieve the above object, a third embodiment of the present invention provides a computer device, which is characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements an air-ground multi-modal multi-agent collaborative positioning and mapping method as described above.

In order to achieve the above object, a fourth embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, which is characterized in that when the computer program is executed by a processor, the above-mentioned open space is implemented Multi-modal multi-agent collaborative localization and mapping method.

The open-ground multi-modal multi-agent collaborative positioning and mapping method proposed by the embodiment of the present invention is suitable for a variety of application scenarios, and significantly improves the perception ability and work efficiency of a single robot system. Especially in the field of search and rescue, aerial drones (Unmanned Aerial Vehicle, UAV) use their aerial perspective to detect unknown environmental terrain, and guide ground robots (Unmanned ground vehicle, UGV) to drive into the target area to carry out precise rescue missions. Search and rescue can be achieved in an efficient and low-cost way compared to manpower. Among them, aerial drones have a global field of view on the ground. Combined with the local environment perception capabilities of ground robots, they can build a globally consistent environmental map model. This map model will provide global navigation information for ground robots.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

Figure 1 is a schematic flowchart of an air-ground multi-modal multi-agent collaborative positioning and mapping method provided by an embodiment of the present invention.

Figure 2 is a schematic diagram of an air-ground multi-modal multi-agent collaborative positioning and mapping system provided by an embodiment of the present invention.

Figure 3 is a schematic diagram of an air-ground multi-modal multi-agent collaborative positioning and mapping device provided by an embodiment of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present invention and are not to be construed as limiting the present invention.

The following describes the air-ground multi-modal multi-agent collaborative positioning and mapping method according to the embodiment of the present invention with reference to the accompanying drawings.

Figure 1 is a schematic flowchart of an air-ground multi-modal multi-agent collaborative positioning and mapping method provided by an embodiment of the present invention.

As shown in Figure 1, the air-ground multi-modal multi-agent collaborative positioning and mapping method includes the following steps:

S101: Obtain the measurement data of the intelligent agent. The intelligent agent includes a drone and an unmanned vehicle. The unmanned vehicle is equipped with visual markers;

Further, in one embodiment of the present invention, obtaining the measurement data of the agent includes:

Preprocess the measurement data, including visual detection and optical flow tracking, and inertial measurement unit IMU pre-integration; where,

Visual detection and optical flow tracking include: applying the least squares method to obtain the velocity vector of the optical flow, as follows:

,

in, ,/> Indicates that the brightness of the pixels in the image is/> ,/> Image gradient in direction, /> Shown in/> Time gradient in direction,/> ,/> is the optical flow edge/> ,/> axis velocity vector;

Inertial measurement unit IMU pre-integration includes:

,

in, is the IMU coordinate system,/> The coordinate system where the IMU is located when initializing the origin is the world coordinate system, /> and/> are the acceleration and angular velocity measured by the IMU,/> for/> Time rotation from IMU coordinate system to world coordinate system,/> It is the right multiplication of the quaternion;

General Frame to/> By integrating all the IMU data between frames, we can get the Frame position/> , speed/> and rotate/> ,/> As an initial value for visual estimation, the rotation is in quaternion form.

Specifically, FAST features are selected for feature extraction and optical flow tracking. In each new image, existing feature points are tracked and new feature points are detected by the KLT algorithm. In order to ensure the uniform distribution of feature points, the image is divided into several sub-regions of the same size, and up to 10 FAST corner points are extracted from each sub-region to keep the number of corner points in each image within a certain range. In outdoor scenes, the displacement between two adjacent frames of images is large and the brightness value of each pixel may suddenly change, which will have a negative impact on the tracking of feature points. Therefore, it is necessary to remove outliers from the feature points and then project them onto the unit sphere. . Outlier elimination uses the RANSAC algorithm to achieve more robust optical flow tracking in outdoor dynamic scenes.

Perform IMU pre-integration while visual detection and tracking. The IMU responds quickly and is not affected by imaging quality. It can estimate the characteristics of absolute scale and supplement the visual positioning of objects with no structure on the outdoor surface. If all poses corresponding to all sampling moments of the IMU are inserted between frames for optimization during camera pose estimation, the program running efficiency will be reduced. Therefore, IMU pre-integration processing is required to convert the high-frequency output acceleration and angular velocity measurement values into A single observation value that will be relinearized in nonlinear iterations to form a constraint factor for the inter-frame state quantity.

S102: Use the measurement data of unmanned vehicles to perform local mapping from a local perspective, and use the measurement data from UAVs to perform local mapping from a global perspective;

The initialization module needs to loosely couple the visual information and IMU information to restore the scale of the monocular camera. First, use SFM to solve the poses of all frames and the three-dimensional positions of all landmark points in the sliding window, and then align them with the previously obtained IMU pre-integrated values to solve for the angular velocity offset, gravity direction, scale factor and each The speed corresponding to one frame. As the system runs, the number of state variables increases, and the sliding window method is used to optimize the state variables within the window.

Further, in one embodiment of the present invention, local perspective mapping is performed using the measurement data of the unmanned vehicle, and local perspective mapping is performed using the measurement data of the unmanned aerial vehicle, including:

Use SFM to solve the poses of all frames and the three-dimensional positions of all landmark points in the sliding window, and align them with the IMU pre-integrated values to obtain the angular velocity offset, gravity direction, scale factor and velocity corresponding to each frame;

Use the sliding window method to optimize the state variables within the window, in Optimized state vector in time window/> As follows,

,

in, and/> is the rotation and translation part of the camera pose,/> is the speed of the camera in the world coordinate system,/> and/> are the acceleration bias and angular velocity bias of the IMU respectively; the optimization objective function of the system's state quantity is as follows:

,

in, is the maximum estimated posterior value,/> is the sliding window initial value residual,/> is the IMU observation residual,/> is the camera observation residual.

S103: Detect the visual markers through the drone. When the detection is successful, obtain the relative pose between the drone and the unmanned vehicle, and use the conversion relationship of the relative pose to perform air-ground perspective map fusion and optimization;

A visual marker is installed above the ground robot. Once the aerial robot observes the marker, it will detect it. Once the detection is successful, the relative pose between the air and ground agents and a corresponding set of key frames will be sent. to the backend.

When the air terminal detects the visual mark carried by the ground robot, the transformation relationship of the visual mark in the coordinate system of the airborne camera is obtained through the visual mark. , assuming the conversion relationship from visual signs to ground robot cameras is/> , then the conversion relationship between the UAV camera coordinate system and the UAV camera coordinate system at the current moment is/> , which is the pose transformation relationship between open and ground key frames at the current moment. At the same time, the key frames generated by the air and ground terminals at the current moment are known. and/> and their respective reference keyframes/> and/> The pose transformation relationship between the maps can be obtained by the pose transformation matrix/> .

S104: Continuously detect whether the agent passes through the overlapping area based on loop closure. When the overlapping area is detected, the association between the two maps of the agent is established through matching key frames, and trajectory calibration and similar perspective map fusion and optimization are performed;

The loop detection method is based on the Bag-of-Words model and uses the DBoW2 library to index the image database in forward and reverse order. The bag-of-word model determines whether a loop is generated by calculating the similarity between the statistical bag-of-word vector and the current frame.

First, all visual features are clustered. One type of feature, that is, a collection of local adjacent feature points, is a "word", so all features are a "dictionary". If you want to have many images in total feature points are classified as/> Class, make "dictionary"into/> Fork tree for storage and query. By determining which "words" in the "dictionary" are present in an image, a vector can be used to describe an image.

Further, in one embodiment of the present invention, continuously detecting whether the agent passes through the overlapping area based on loop closure includes:

All visual features are clustered, one type of feature is a word, and all features are a dictionary;

Describe an image with a single vector:

Calculate the similarity between two images A and B :

,

in, is a vector describing image A, /> is a vector describing image B, /> is the similarity between two images A and B, /> Represents a vector describing image A /> of/> components,/> represents a vector describing image B of/> weight;

If the similarity exceeds the threshold, a loop is considered to have occurred.

Further, in one embodiment of the present invention, all visual features are clustered, including:

At the root node, use the K-means algorithm to cluster all samples into Class, get the first layer;

For each node in the first layer, the samples belonging to the node are clustered into Class, get the next layer; and so on, finally get the leaf layer, which is the word.

Further, in one embodiment of the present invention, a single vector is used to describe an image, including:

definition for word/> The number of features included,/> is the number of features contained in all words,/> for words The number of times it appears in image A,/> for word/> The total number of occurrences in all images, then

word weights in image A/> for:/> ,

via bag-of-words, using a single vector Describe an image A:

,

in, is the word that image A has in the dictionary,/> yes The corresponding weight,/> is a vector describing image A.

S105: Obtain a map with pose trajectory and global consistency based on the map fused and optimized from the air-ground perspective and the map fused and optimized from the similar perspective.

The map fusion and optimization module is divided into two parts: the similar perspective and the open-ground perspective. The open-ground perspective will only be executed when the back-end receives information that the visual mark detection is successful. It will use the relative pose relationship to directly convert the matching open-ground end map. , fusion and optimization;

Similar viewing angles are always in progress. The newly stored key frames continuously detect whether the sub-end passes through the overlapping area based on the loop. The overlapping area includes the sub-end's own loop area and the overlapping area between the sub-ends. Once it is detected that the areas passed by the two terminals overlap, the association of the two maps is established through the matching key frames, the coordinate conversion relationship is obtained, and the maps are further fused. After the map fusion, the two maps are merged, and these maps will be are removed from the map stack, and a new global map resulting from their fusion will be added to the map stack and directly associated with both subends.

After the backend is optimized, the optimized pose will be sent to the corresponding sub-end. The sub-end will update the pose and use it as a constraint in the pose graph for local optimization, thereby optimizing its own local map to perform more accurately. The subsequent positioning and mapping process.

The final output pose trajectory and globally consistent map.

Figure 2 is a schematic diagram of an air-ground multi-modal multi-agent collaborative positioning and mapping system proposed by an embodiment of the present invention, which is divided into two parts: a sub-end and a back-end. UAVs and unmanned vehicles are both called agents. Each agent runs a sub-end independently. The sub-end is equipped with a camera, IMU, gimbal and its control system, as well as communication units and boards that can exchange data with the central server. The onboard processing unit runs an independent front-end visual inertial odometry, sends key frames and map points to the back-end and maintains a smaller local map; the back-end receives information from each sub-end and performs visual marker detection through the central server. , loop closure detection, fusion optimization and other computationally intensive operations, output pose trajectories, and create a globally consistent map. The system uses ROS for communication between the sub-end and the back-end. The sub-end transmits the captured key frames and map points to the back-end, and the back-end transmits the updated pose to the sub-end.

The present invention proposes an open-ground multi-modal multi-agent collaborative positioning and mapping method, which is suitable for a variety of application scenarios and significantly improves the perception ability and work efficiency of a single robot system. Especially in the field of search and rescue, aerial drones (Unmanned Aerial Vehicle, UAV) use their aerial perspective to detect unknown environmental terrain, and guide ground robots (Unmanned ground vehicle, UGV) to drive into the target area to carry out precise rescue missions. Search and rescue is achieved in an efficient and low-cost way compared to manpower. Among them, aerial drones have a global field of view on the ground. Combined with the local environment perception capabilities of ground robots, they can build a globally consistent environmental map model. This map model will provide ground robots with necessary information for global navigation.

Compared with the existing technology, the advantages of the present invention are:

1) Pure visual SLAM has some shortcomings. Vision has good ability to capture texture features in the scene, but it has insufficient ability to capture structural features in the environment. It is also sensitive to initialization and lighting, and the monocular camera cannot obtain pose and map information. Absolute scale. On the basis of visual detection and tracking, this application adds an IMU sensor to perform multi-modal information fusion. The IMU responds quickly and is not affected by imaging quality. It can estimate the characteristics of absolute scale and perform visual positioning of objects with no structure on outdoor surfaces. Replenish.

2) The vibration generated during the movement of the intelligent body will be transmitted to the camera along with the fixed structure, which will affect the collection of feature points by the front-end system. This application introduces a gimbal and its control system to the camera to achieve the purpose of anti-shake and anti-shock.

In order to implement the above embodiments, the present invention also proposes an air-ground multi-modal multi-agent collaborative positioning and mapping device.

Figure 3 is a schematic diagram of an air-ground multi-modal multi-agent collaborative positioning and mapping device provided by an embodiment of the present invention.

As shown in Figure 3, the air-ground multi-modal multi-agent collaborative positioning and mapping device includes: an acquisition module 100, a mapping module 200, an air-ground perspective fusion module 300, a similar perspective fusion module 400, and an output module 500, where,

The acquisition module is used to obtain measurement data of intelligent agents. Intelligent agents include drones and unmanned vehicles. The unmanned vehicles are equipped with visual markers;

The mapping module is used to perform local mapping from a local perspective using the measurement data of unmanned vehicles, and perform local mapping from a global perspective using measurement data from UAVs;

The air-ground perspective fusion module is used to detect visual markers through drones. When the detection is successful, the relative pose between the drone and the unmanned vehicle is obtained, and the relative pose conversion relationship is used to perform air-ground perspective map fusion and optimization;

The similar perspective fusion module is used to continuously detect whether the agent passes through the overlapping area based on loop closure. When the overlapping area is detected, the association between the two maps of the agent is established through matching key frames to perform trajectory calibration and similar perspective map fusion and optimization. ;

The output module is used to obtain a map with pose trajectory and global consistency based on the map fused and optimized from the air-ground perspective and the map fused and optimized from the similar perspective.

In order to achieve the above object, a third embodiment of the present invention provides a computer device, which is characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the air-ground multi-modal multi-agent collaborative positioning and mapping method as described above is implemented.

In order to achieve the above object, a fourth embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. It is characterized in that when the computer program is executed by a processor, the above-mentioned air-ground multi-mode is implemented. State-of-the-art multi-agent collaborative localization and mapping method.

In the description of this specification, reference to the terms "one embodiment," "some embodiments," "an example," "specific examples," or "some examples" or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the invention. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine different embodiments or examples and features of different embodiments or examples described in this specification unless they are inconsistent with each other.

In addition, the terms “first” and “second” are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically limited.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above-mentioned embodiments are illustrative and should not be construed as limitations of the present invention. Those of ordinary skill in the art can make modifications to the above-mentioned embodiments within the scope of the present invention. The embodiments are subject to changes, modifications, substitutions and variations.

Claims (8)

1. An air-ground multi-modal multi-agent collaborative positioning and mapping method, which is characterized by including the following steps: Obtaining measurement data of an intelligent agent, the intelligent agent includes a drone and an unmanned vehicle, and the unmanned vehicle is provided with a visual marker; Local mapping from a local perspective is performed using the measurement data of the unmanned vehicle, and local mapping from a global perspective is performed using the measurement data of the UAV; The visual marker is detected by the drone, and when the detection is successful, the relative pose between the drone and the unmanned vehicle is obtained, and the conversion relationship between the relative pose is used to perform an air-ground perspective. Map fusion and optimization; Continuously detect whether the agent passes through the overlapping area based on loop closure. When the overlapping area is detected, the association between the two maps of the agent is established through matching key frames, and trajectory calibration and similar perspective map fusion and optimization are performed; According to the map after the fusion and optimization of the air and ground perspectives and the map after the fusion and optimization of the similar perspective, a map with pose trajectory and global consistency is obtained; Wherein, obtaining the measurement data of the intelligent agent includes: Preprocess the measurement data, including visual detection and optical flow tracking, and inertial measurement unit IMU pre-integration; where, The visual detection and optical flow tracking include: applying the least squares method to obtain the velocity vector of the optical flow, as follows: , in, ,/> Indicates that the brightness of the pixels in the image is/> ,/> Image gradient in direction, /> Shown in/> Time gradient in direction,/> ,/> is the optical flow edge/> ,/> axis velocity vector; The inertial measurement unit IMU pre-integration includes: , in, is the IMU coordinate system,/> The coordinate system where the IMU is located when initializing the origin is the world coordinate system, /> and/> are the acceleration and angular velocity measured by the IMU,/> for/> Time rotation from IMU coordinate system to world coordinate system,/> It is the right multiplication of the quaternion; General Frame to/> By integrating all the IMU data between frames, we can get the Frame position/> , speed/> and rotate/> ,/> As an initial value for visual estimation, the rotation is in quaternion form. 2. The method according to claim 1, characterized in that the local perspective mapping is performed using the measurement data of the unmanned vehicle, and the global perspective local mapping is performed using the measurement data of the unmanned aerial vehicle, including : Use SFM to solve the poses of all frames and the three-dimensional positions of all landmark points in the sliding window, and align them with the IMU pre-integrated values to obtain the angular velocity offset, gravity direction, scale factor and velocity corresponding to each frame; Use the sliding window method to optimize the state variables within the window, in Optimized state vector in time window/> As follows, , in, and/> is the rotation and translation part of the camera pose,/> is the speed of the camera in the world coordinate system, and/> are the acceleration bias and angular velocity bias of the IMU respectively; the optimization objective function of the system's state quantity is as follows: , in, is the maximum estimated posterior value,/> is the sliding window initial value residual,/> is the IMU observation residual,/> is the camera observation residual. 3. The method according to claim 1, characterized in that the continuous detection of whether the agent passes through an overlapping area based on loop closure includes: All visual features are clustered, one type of feature is a word, and all features are a dictionary; Describe an image with a single vector: Calculate the similarity between two images A and B : , in, is a vector describing image A, /> is a vector describing image B, /> is the similarity between two images A and B, /> Represents a vector describing image A /> of/> components,/> Represents a vector describing image B/> of/> weight; If the similarity exceeds the threshold, a loop is considered to have occurred. 4. The method of claim 3, wherein clustering all visual features includes: At the root node, use the K-means algorithm to cluster all samples into Class, get the first layer; For each node in the first layer, the samples belonging to the node are clustered into Class, the next layer is obtained; and so on, finally the leaf layer is obtained, and the leaf layer is the word. 5. The method according to claim 3, characterized in that describing an image with a single vector includes: definition for word/> The number of features included,/> is the number of features contained in all words,/> for word/> The number of times it appears in image A,/> for word/> The total number of occurrences in all images, then word weights in image A/> for:/> , via bag-of-words, using a single vector Describe an image A: , in, is the word that image A has in the dictionary,/> yes The corresponding weight,/> is a vector describing image A. 6. An air-ground multi-modal multi-agent collaborative positioning and mapping device, which is characterized by including the following modules: An acquisition module, used to acquire measurement data of an intelligent agent. The intelligent agent includes a drone and an unmanned vehicle, and the unmanned vehicle is provided with a visual marker; A mapping module, configured to perform local mapping from a local perspective using the measurement data of the unmanned vehicle, and perform local mapping from a global perspective using the measurement data of the drone; An air-ground perspective fusion module is used to detect the visual marker through the drone, and when the detection is successful, obtain the relative pose between the drone and the unmanned vehicle, using the relative pose Perform air-ground perspective map fusion and optimization based on the posture conversion relationship; A similar perspective fusion module is used to continuously detect whether the agent passes through an overlapping area based on loop closure. When the overlapping area is detected, the association between the two maps of the agent is established through matching key frames, and trajectory calibration and Similar perspective map fusion and optimization; An output module, configured to obtain a map with pose trajectory and global consistency based on the map after fusion and optimization of the air-ground perspective and the map after fusion and optimization of the similar perspective; Wherein, obtaining the measurement data of the intelligent agent includes: Preprocess the measurement data, including visual detection and optical flow tracking, and inertial measurement unit IMU pre-integration; where, The visual detection and optical flow tracking include: applying the least squares method to obtain the velocity vector of the optical flow, as follows: , in, ,/> Indicates that the brightness of the pixels in the image is/> ,/> Image gradient in direction, /> Shown in/> Time gradient in direction,/> ,/> is the optical flow edge/> ,/> axis velocity vector; The inertial measurement unit IMU pre-integration includes: , in, is the IMU coordinate system,/> The coordinate system where the IMU is located when initializing the origin is the world coordinate system, /> and/> are the acceleration and angular velocity measured by the IMU,/> for/> Time rotation from IMU coordinate system to world coordinate system,/> It is the right multiplication of the quaternion; General Frame to/> By integrating all the IMU data between frames, we can get the Frame position/> , speed/> and rotate/> ,/> As an initial value for visual estimation, the rotation is in quaternion form. 7. A computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the following: The air-ground multi-modal multi-agent collaborative positioning and mapping method described in any one of requirements 1-5. 8. A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the air-ground multi-modal multi-agent according to any one of claims 1-5 is implemented. Co-localization and mapping methods.