CN111652261A - A Multimodal Perception Fusion System - Google Patents

Abstract

The present invention provides a multi-modal perception fusion system for a full scene. The multi-modal perception fusion system includes a host computer, a laser radar, a multi-eye camera, an IMU, an infrared depth camera, and a power supply, and the multi-eye camera includes Two FLIR industrial network port cameras and two USB3.0 cameras, the steps to form a multi-modal perception fusion system are: installation of hardware, software installation, data acquisition, and model construction. The present invention uses the McQuarte algorithm in the modeling identification to iteratively optimize the external parameters to obtain the optimal estimation, thereby obtaining the most accurate model and effect diagram, making the fusion more accurate, realizing the real-time perception environment, and the multi-modality The perception fusion system is small and light in weight, and can be used in the modeling of unmanned vehicles, unmanned aerial vehicles, medical industry, and military unmanned environments. It can also be used in various complex environments such as indoor and outdoor, laying the foundation for planning and navigation.

Description

A Multimodal Perception Fusion System

technical field

The invention belongs to the field of multimodal perception fusion systems, and in particular relates to a multimodal perception fusion system for full scenes.

Background technique

With the rapid development of sensor technology and the Internet, various modalities of big data are rapidly emerging at an unprecedented development speed. For a thing to be described (target, scene, etc.), the coupled data samples collected through different methods or perspectives are multimodal data, and each method or perspective for collecting these data is usually called a modality.

Narrow multimodal information usually focuses on modalities with different perceptual characteristics, while generalized multimodal fusion usually also includes multi-feature fusion in the same modal information, and data fusion of multiple sensors of the same type. The problem of modal perception and learning is closely related to "multi-source fusion" and "multi-sensor fusion" in the field of signal processing, and "multi-view learning" or "multi-view fusion" in the field of machine learning; The data can obtain more comprehensive and accurate information, and enhance the reliability and fault tolerance of the system.

In the multimodal perception and learning problem, since different modalities have completely different description forms and complex coupling correspondences, it is necessary to uniformly solve the problems of multimodal perception representation and cognitive fusion. Multimodal perception and fusion is to make two seemingly unrelated data samples of different formats, through appropriate transformation or projection, to be compared and fused with each other. This kind of fusion of heterogeneous data can often achieve unexpected results.

At present, multimodal data has played a huge role in Internet information search, human-computer interaction, industrial environment fault diagnosis and robotics. In the field of robotics, there are still many challenging problems that need to be further explored; therefore, we have developed a multi-modal perception system that integrates multi-modal vision, laser, binocular infrared, depth, IMU and other multi-modalities. state, these hardwares are installed in different orientations. In order to realize the automatic perception, scanning and modeling of large scenes and small workpieces, it can realize the perception of the whole scene, suitable for indoor and outdoor, and give depth information and distance information to the RGB image information of the environment, but the main difficulty is It lies in: heterogeneous multi-source sensors, feature extraction, and the solution of correlation between features make the fusion more accurate and can achieve real-time perception of the environment.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned technical problems, the present invention provides a multi-modal perception fusion system for the whole scene, so as to realize the automatic perception, scanning and modeling of large-scale scenes and small workpieces. The multi-modal perception fusion system includes a host computer, a lidar, a multi-eye camera, an IMU, an infrared depth camera, and a power supply. The multi-eye camera includes two FLIR industrial network port cameras and two USB3.0 cameras. The steps of the multimodal perception fusion system are:

S1: Install hardware: connect the lidar to the host computer with the Ethernet interface, connect the two FLIR industrial network port cameras to the host computer with the Ethernet interface, connect the two USB3.0 cameras, IMU and infrared depth The camera is respectively connected to the usb3.0 interface of the host computer, and after all parts are connected, it is connected to the power supply through the data cable;

S2: Software installation and data acquisition: Open the Linux Ubuntu system, install and configure the drivers and software of each module, use the Robot Operating System to start the nodes of each mode, and use the RVIZ to obtain the point cloud, multi-modality, and multi-mode of the lidar. The RGB image of the camera, the accelerometer and gyroscope information of the IMU, and the depth map of the infrared depth camera are all displayed;

S3: Model construction: Then use the SLAM theoretical system to process the acquired data. The processing flow is divided into two steps, namely the front-end and the back-end. The front-end is responsible for the feature extraction of each module and the representation of the correlation between features. The back-end is responsible for parameter optimization, 3D reconstruction and positioning, and iteratively optimizes the external parameters using the McQuarter algorithm in modeling identification to obtain the optimal estimate, thereby obtaining the final model and rendering of the fusion.

Preferably, the operating system used by the multimodal perception fusion system is Linux Ubuntu system, the middleware used is Robot Operating System, and the programming languages used are c++ and python.

Preferably, the laser radar adopts Leishen Intelligence c16-151B.

Preferably, the number of the infrared depth cameras is two, and the infrared depth cameras and the IMU adopt IntelReal Sense D435i.

Preferably, the projected distance of the lidar to the ground is 10m, and there will be a cone-shaped blind spot below after projection, and the working distance of the infrared depth camera is 0.2-10m, which can make up for the blind spot that cannot be projected by the lidar.

Preferably, the lidar, multi-eye camera, IMU, and infrared depth camera all have independent sensors.

Compared with the prior art, the beneficial effects of the present invention are: autonomous combination of heterogeneous sensors, rapid calibration, matching of collected information and fusion in three-dimensional space, generating a patch model from point clouds, and then performing iterative optimization, Finally, a 3D reconstruction model that can achieve accuracy is obtained, so as to obtain the most accurate model and renderings, which makes the fusion more accurate, can achieve real-time perception of the environment, and provide accurate technical data for future identification and detection technology. The fusion system is small and light in weight, and can be used in the modeling of unmanned vehicles, unmanned aerial vehicles, medical industry, and military unmanned environments. It can also be used in various complex environments such as indoor and outdoor, laying the foundation for planning and navigation.

Description of drawings

Figure 1 is an appearance diagram of a full-scene multimodal perception fusion system.

Fig. 2 is the multimodal system structure diagram of the whole scene;

Fig. 3 is a diagram showing the installation steps of a full-scene multimodal perception fusion system.

In the picture: 1- Lidar; 2- The first FLIR industrial network port camera; 3- The first USB3.0 camera; 4- The first infrared depth camera; 5- The second infrared depth camera; 6- The second FLIR industrial network mouth camera; 7-second USB3.0 camera; 8-multi-eye camera; 9-IMU.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

The present invention is further described below:

Example:

As shown in accompanying drawing 1, a kind of multimodal perception fusion system for the whole scene, the operating system adopted by the multimodal perception fusion system is the Linux Ubuntu system, the middleware used is the Robot Operating System, and the programming used is The languages are c++ and python; the multi-modal perception fusion system includes a host computer, a lidar 1, a first FLIR industrial network port camera 2, a second FLIR industrial network port camera 6, a first USB3.0 camera 3, a second camera USB3.0 camera 7, multi-eye camera 8, IMU 9, first infrared depth camera 4, second infrared depth camera 5, power supply;

Specifically, as shown in Figure 3, the steps of forming a multimodal perception fusion system are:

S1: Install the hardware: connect the lidar to the host computer via the Ethernet interface, connect the first FLIR camera 2 and the second FLIR camera 3 to the host computer via the Ethernet interface, and connect the first FLIR camera 2 and the second FLIR camera 3 to the host A USB3.0 camera 3, a second USB3.0 camera 7, an IMU 9, and the first infrared depth camera 4 and the second infrared depth camera 5 are respectively connected to the usb3.0 interface of the host computer, and after all parts are connected, the data is passed through The line is connected to the power supply;

S2: Software installation and data acquisition: Open the Linux Ubuntu system, install and configure the drivers and software of each module, use the Robot Operating System to start the nodes of each mode, and use RVIZ to obtain the point cloud of lidar 1, Multi-eye camera 8 , the first FLIR industrial port camera 2 , the second FLIR industrial port camera 6 , the RGB images of the first USB3.0 camera 3 and the second USB3.0 camera 7 , the accelerometer and gyroscope of the IMU 9 Information and these data of the depth map of the first infrared depth camera 4 and the second infrared depth camera 5 are displayed;

S3: Model construction: Then use the slam theoretical system to process the acquired data. The processing flow is divided into two steps, namely the front-end and the back-end. The front-end is responsible for the feature extraction of each module and the representation of the correlation between features. The back-end is responsible for parameter optimization, 3D reconstruction and positioning, and iteratively optimizes external parameters using the McQuarte algorithm in modeling identification to obtain the optimal estimate, and finally obtains the final model and rendering after accurate fusion.

Specifically, the laser radar 1 adopts Leishen Intelligence c16-151B.

Specifically, the first infrared depth camera 4, the second infrared depth camera 5 and the IMU 9 all use IntelReal Sense D435i.

Specifically, the projected distance of the lidar 1 to the ground is 10m, and there will be a cone-shaped blind spot below after projection. The working distance of the first infrared depth camera 4 and the second infrared depth camera 5 is 0.2-10m, which can make up for The blind spot that lidar 1 can't cast.

Specifically, the lidar 1 , the first FLIR industrial network port camera 2 , the second FLIR industrial network port camera 6 , the first USB3.0 camera 3 , the second USB3.0 camera 7 , the multi-eye camera 8 , and the IMU 9 , the first infrared depth camera 4 and the second infrared depth camera 5 have independent sensors respectively.

As shown in FIG. 2, a diagrammatic representation of a multimodal system is shown. Among them, the vertices represent sensors such as lidar, camera, and IMU. Edges represent relative pose transformation derivations between sensors.

The work flow chart is shown in Figure 3.

It should be noted that, in this document, also, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not expressly listed or inherent to such a process, method, article or apparatus.

Although the embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications and substitutions can be made in these embodiments without departing from the principle and spirit of the invention and modifications, the scope of the present invention is defined by the appended claims and their equivalents.

Claims (7)

1. The multi-modal perception fusion system for the full scene is characterized by comprising an upper computer, a laser radar, a multi-view camera, an IMU (inertial measurement Unit), an infrared depth camera and a power supply, wherein the multi-view camera comprises two FLIR (flash infrared) industrial network port cameras and two USB (universal serial bus) 3.0 cameras, and the multi-modal perception fusion system comprises the following steps:

s1: installing hardware: connecting a laser radar to an upper computer in an Ethernet interface connection mode, connecting two FLIR industrial network port cameras to the upper computer in the Ethernet interface mode, respectively connecting two USB3.0 cameras, an IMU (inertial measurement unit) and an infrared depth camera to a USB3.0 interface of the upper computer, and connecting all the parts after being connected with each other through a data line and a power supply;

s2: installation of software and acquisition of data: opening a Linux Ubuntu System, installing and configuring a driver and software of each module, starting nodes of each mode by using a Robot Operating System, and displaying the acquired data of the point cloud of the laser radar, the RGB image of the multi-view camera, the information of an accelerometer and a gyroscope of the IMU and the depth-of-field map of the infrared depth camera by using RVIZ;

s3: constructing a model: and then, processing the acquired data by using a slam theoretical system, wherein the processing flow is divided into two steps, namely a front end and a rear end, the front end is responsible for feature extraction of each module and representation of correlation among features, and the rear end is responsible for optimization, three-dimensional reconstruction and positioning of parameters, so that a fused final model and an effect graph are obtained finally.

2. The System according to claim 1, wherein the operating System adopted by the multimodal perceptual fusion System is Linux Ubuntu System, the middleware adopted by the multimodal perceptual fusion System is RobotOperating System, and the programming languages used by the multimodal perceptual fusion System are c + + and python.

3. The multi-modal perceptual fusion system for a whole scene of claim 1 wherein the lidar employs radium intelligence c 16-151B.

4. The multi-modal perceptual fusion system for a full scene of claim 1 wherein the number of infrared depth cameras is two and the infrared depth cameras and IMU employ Intel Real sensor D435 i.

5. The multi-modal perceptual fusion system for a full scene as claimed in claim 1 wherein the final model and effect map is obtained by using a Marquardt algorithm in modeling recognition to iteratively optimize the external parameters to obtain an optimal estimation, thereby obtaining the most accurate model and effect map.

6. The system of claim 1, wherein the distance from the laser radar projected to the ground is 10m, a conical blind area is formed below the projected laser radar, and the working distance of the infrared depth camera is 0.2-10m, so that the blind area which cannot be projected by the laser radar can be compensated.

7. The multi-modal perceptual fusion system for a whole scene of claim 1 wherein the lidar, the multi-view camera, the IMU, and the infrared depth camera each have independent sensors.

Images