CN113408625B - Multi-source heterogeneous data single-frame fusion and consistent characterization method applied to unmanned system - Google Patents

Abstract

The invention discloses a multi-source heterogeneous data single-frame fusion and consistency characterization method applied to an unmanned system. The method can complete the fusion of single-frame heterogeneous data generated by a plurality of sensors and the consistent representation of the environment, and realize the comprehensive perception and description of the environment. The method can effectively solve the problems of effective fusion of multi-source heterogeneous data generated by various sensors and consistent representation of detection results.

Description

Single-frame fusion and consistent representation of multi-source heterogeneous data for unmanned systems

technical field

The invention relates to the technical field of sensor data fusion, in particular to a single frame fusion and consistent representation method of multi-source heterogeneous data applied to an unmanned system.

Background technique

The unmanned system uses heterogeneous data generated by a variety of different sensors, including but not limited to 2D image data, 3D point cloud data, inertial navigation data, astronomical data, temperature data, mechanical data and other multi-source heterogeneous data to complete data fusion. As well as a consistent representation of the environment, it can maximize the use of data generated by a variety of different sensors to achieve comprehensive perception and description of the environment. Traditional methods are mainly based on feature-level and decision-level fusion, and there are few methods for data-level fusion among a large number of heterogeneous data, especially for single-frame data-level fusion, which can provide more comprehensive and higher-quality unmanned systems. perceptual information.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a multi-source heterogeneous data single-frame fusion and consistent characterization method applied to unmanned systems, to realize the fusion of heterogeneous data of different sensors, to complete the consistent characterization of the environment, and to solve the difficulty of multi-source heterogeneous data. The problem of data-level fusion.

In order to achieve the above object, technical scheme of the present invention is as follows:

A multi-source heterogeneous data single-frame fusion and consistent characterization method applied to an unmanned system, the method includes the following steps:

Step 1: Data collection and preprocessing;

Use M multi-source sensors to collect data at the same time, and preprocess the obtained M data blocks, so that the M data blocks have descriptions of three-dimensional space position attributes, and the three-dimensional space position attributes share the same three-dimensional space coordinates Tie;

Step 2: Fusion of the same type of data;

Classify the preprocessed M data blocks according to the sensor type, classify the data blocks with the same sensor principle or characterize the same physical properties as the same type of data, and perform data-level fusion on the same type of data to obtain N isomorphic fusions The fusion data set of N is the number of heterogeneous types of heterogeneous data;

Step 3: Align the dimensions of the fusion data set;

Perform dimension statistics on N fusion data groups, and construct a data expression model of X dimension, where X is the minimum value that can cover all different dimensions of each fusion data group; map the N fusion data groups to this data expression model, and output N A dimension aligned data set;

Step 4: Heterogeneous fusion and consistent characterization;

Analyze the data expression model to obtain redundant relationships and dimension transformation relationships in X dimensions, select the final data form to be output according to the actual task, form a consistent representation data space with corresponding dimensions, and align the N dimensions The latter data set is transformed into this data space according to the dimension transformation relationship, and after processing by confidence discrimination, it is merged into consistent representation data and output.

Further, the M multi-source sensors in the step 1 refer to M sensors that provide data sources, and the multi-source heterogeneous data types collected by them include 2D image data, 3D point cloud data, inertial navigation data, and astronomical data. , temperature data, mechanical data.

Further, the preprocessing in the first step includes any one or more of smoothing denoising, missing value processing, data normalization, and chromatic aberration correction. and sensor parameter model to estimate the three-dimensional spatial position of the collected data and output it as a fixed attribute of the data.

Further, the data-level fusion in the second step is a merging process of the preprocessed data, including redundancy elimination and data normalization.

Further, the data expression model in the third step refers to a data expression space that includes attribute dimensions of all the data to be fused, and is a union of the original attribute dimensions of all the data to be fused.

Further, the dimension transformation relationship in the step 4 refers to the data conversion method between the original data set and the new data set when the original data is changed to the data dimension. The data conversion method can change the form and description method of the original data, but The integrity of the data information should be maintained as much as possible, and the data transformation method includes any one of grid generation, principal component analysis, factor analysis, linear combination, and clustering.

Further, the confidence discrimination processing in the step 4 is a processing strategy for multi-source heterogeneous data when a conflict occurs at the same data space point after being converted to the same data description, and the confidence parameter includes a priori-based data source priority. , the original data error statistics, data density, and data consistency in the preprocessing, and finally the conflict-merged data is obtained by weighted calculation according to the confidence parameter.

The beneficial effects of the present invention are as follows:

Through data collection and preprocessing, the same type of data fusion, the dimensional alignment of fusion data groups, and the fusion and consistent representation of heterogeneous data, the invention solves the fusion difficulties caused by the problems of different meanings and different dimensions of multi-source heterogeneous data. It realizes data-level fusion and consistent representation between single-frame heterogeneous data.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

FIG. 1 is a flow chart of the method for single frame fusion and consistent characterization of multi-source heterogeneous data provided by the present invention.

Fig. 2 is a flow chart of a method for single-frame fusion and consistent characterization of multi-source heterogeneous data according to an exemplary embodiment.

Detailed ways

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

As shown in FIG. 1 , the method for single-frame fusion and consistent characterization of multi-source heterogeneous data applied to an unmanned system of the present invention, the specific implementation steps are as follows:

Step 1: Data acquisition and preprocessing, using M multi-source sensors to collect data at the same time, and preprocessing the obtained M data blocks, so that the M data blocks have descriptions of three-dimensional spatial position attributes, and the three-dimensional Spatial location attributes share the same three-dimensional spatial coordinate system.

Further, in conjunction with the description of an embodiment shown in FIG. 2 , the following sub-steps are used to achieve:

S101: M multi-source sensors are fixedly installed on the unmanned system, which share the same three-dimensional space coordinate system. Using a combination of hardware triggering and software synchronization to complete the data acquisition of M multi-source sensors, so that the time consistency accuracy of all data reaches the millisecond level;

Specifically, M multi-source sensors refer to M sensors that provide data sources, and the sensor types may be the same or different, and the multi-source heterogeneous data types collected by them include but are not limited to 2D image data, 3D point cloud data, Inertial navigation data, astronomical data, temperature data, mechanical data. Use the lower computer MCU to synchronously generate multiple PWM pulse signals to trigger the corresponding sensor, and the sensor starts to collect data on the rising or falling edge of the PWM signal. At the same time, the software compensation method is used to compensate the time delay of sensor data in the signal transmission link, so that the upper computer can obtain a frame of multi-source heterogeneous data with the same time stamp at the same time.

S102: Complete multi-source heterogeneous data preprocessing;

Specifically, preprocessing includes, but is not limited to, smoothing and denoising the collected multi-source heterogeneous data, missing value processing, data normalization, chromatic aberration correction, etc.; It is necessary to estimate the three-dimensional space position of the collected data according to the sensor installation position and the sensor parameter model, and output the three-dimensional space position information as a fixed attribute of the data.

Step 2: Fusion of the same type of data;

Classify the preprocessed M data blocks according to the sensor type, classify the data blocks with the same sensor principle or characterize the same physical properties as the same type of data, and perform data-level fusion on the same type of data to obtain N isomorphic fusions. The fusion data set of N is the number of heterogeneous types of heterogeneous data.

Further, in conjunction with the description of an embodiment shown in FIG. 2 , the following sub-steps are used to achieve:

Through the steps of feature extraction, feature description, feature matching, association calculation, affine transformation, etc., the 2D images from different sensors are used to achieve alignment of multiple 2D images and fusion at the pixel level, and remove redundant parts of the image. output in the form of an image;

The 3D point cloud data from different sensors, through the steps of point cloud feature extraction, point cloud registration, point cloud fusion, etc., completes the data-level fusion between the single-frame point cloud data of multiple sensors, and eliminates redundant information, The final output is in the form of a 3D point cloud;

The sensory data generated by other sensors such as inertial navigation data, astronomical data, temperature data, and mechanical data from multiple different sensors are mutually verified, merged, fused, and redundantly eliminated, and finally output in the form of data.

Step 3: Align the dimensions of the fusion data set.

Based on the dimensional statistics of N fusion data sets, construct a data expression model of X dimension, where X is the minimum value that can cover all different dimensions of each fusion data set; map the N fusion data sets to this data expression model, and output N A dimension-aligned dataset.

Further, in conjunction with the description of an embodiment shown in FIG. 2 , the following sub-steps are used to achieve:

Step S301: In this embodiment, the data dimension of the 3D point cloud is three-dimensional (xyz), the data dimension of the 2D image is five-dimensional (pixel coordinates xy+color information RGB), and the thermal infrared data dimension is three-dimensional (pixel coordinates xy+degree Celsius 1). ), the data dimension of mechanics is a four-dimensional value (sensor coordinates xyz + Newton N), the data dimension of astronomical data is a two-dimensional angle value (degrees), and the inertial navigation data is six-dimensional (velocity, angular rate and acceleration in three directions), A total of 18-dimensional data, including:

[3dx,3dy,3dz,2dx,2dy,R,G,B,I,N,d1,d2,gx,gy,gz,ax,ay,az].

Step S302 : forming a frame of multi-sensor data into an 18-dimensional vector, the position of the sensor without valid data in the vector is empty, and each frame of data becomes a vector with the same length.

Step 4: Heterogeneous fusion and consistent characterization.

Analyze the data expression model to obtain the redundant relationship and dimension transformation relationship in the X dimensions, select the final data form to be output according to the actual task, form a consistent representation data space with corresponding dimensions, and align the N dimensions. The data group is transformed into this data space according to the dimensional transformation relationship, and merged into consistent representation data after processing by confidence discrimination.

Further, in conjunction with the description of an embodiment shown in FIG. 2 , the following sub-steps are used to achieve:

Step S401: Perform dimension reduction on the data vector in step S302, such as principal component analysis, and combine external parameter information such as the installation position of each sensor to obtain the redundancy relationship and dimension transformation relationship between the 18-dimensional data in step S301.

Step S402: According to the output requirements of the visualization task, you can choose to directly output the 2D image data; or use the 3D point cloud data output in the step as the base, and perform grid reconstruction, grid correction and other steps on the data, and the output 2D image data. The data map is on the 3D point cloud to complete the output of 3D data; or other data is superimposed on the above-mentioned 3D data in the form of layered rendering to form the output of multi-source heterogeneous fusion data, and form a consistent representation result with corresponding dimensions .

The embodiment of the present invention described above in conjunction with an embodiment is only a preferred embodiment of the present invention. Of course, this does not limit the scope of rights of the present invention. Those of ordinary skill in the art can understand all or part of the process for realizing the above example. , and the equivalent changes made according to the claims of the present invention still belong to the scope covered by the present invention.

Claims (7)

1. A multi-source heterogeneous data single-frame fusion and consistent characterization method applied to an unmanned system is characterized by comprising the following steps:

the method comprises the following steps: data acquisition and pretreatment;

carrying out simultaneous data acquisition by using M multi-source sensors, and preprocessing the obtained M data blocks to ensure that the M data blocks all have the description of the three-dimensional spatial position attribute, and the three-dimensional spatial position attributes share the same three-dimensional spatial coordinate system;

step two: fusing data of the same type;

classifying the preprocessed M data blocks according to the sensor types, classifying the data blocks with the same sensor principle or representing the same physical property into the same type of data, and performing data level fusion on the same type of data to obtain N isomorphic fused data groups, wherein N is the heterogeneous type quantity of heterogeneous data;

step three: dimension alignment of the fused data sets;

carrying out dimension statistics on the N fusion data groups, and constructing an X-dimension data expression model, wherein X is the minimum value capable of covering all different dimensions of each fusion data group; mapping the N fusion data sets to the data expression model, and outputting N data sets with aligned dimensions;

step four: heterogeneous fusion and consistent characterization;

and analyzing the data expression model to obtain a redundant relation and a dimension transformation relation in X dimensions, selecting a data form which needs to be output finally according to an actual task to form a consistent characterization data space with corresponding dimensions, converting the data set with the aligned N dimensions into the data space according to the dimension transformation relation, combining the data set into consistent characterization data after confidence discrimination processing, and outputting the consistent characterization data.

2. The method for single-frame fusion and consistent characterization of multi-source heterogeneous data applied to an unmanned system according to claim 1, wherein the M multi-source sensors in the first step are M sensors providing data sources, and the types of the multi-source heterogeneous data acquired by the first step include 2D image data, 3D point cloud data, inertial navigation data, astronomical data, temperature data and mechanical data.

3. The method for fusing and uniformly characterizing the single multi-source heterogeneous data frame applied to the unmanned system according to claim 1, wherein the preprocessing in the first step includes any one or more of smoothing denoising, missing value processing, data normalization and chromatic aberration correction, and for a data source not including three-dimensional position information, the three-dimensional spatial position of the acquired data is further estimated according to a sensor installation position and a sensor parameter model and output as a fixed attribute of the data.

4. The method for fusing and uniformly characterizing the single multi-source heterogeneous data frame applied to the unmanned system according to claim 1, wherein the data-level fusion in the second step is a merging process of preprocessed data, and the merging process comprises redundancy elimination and data normalization.

5. The method according to claim 1, wherein the data expression model in step three is a data expression space containing attribute dimensions of all data to be fused, and is a union of original attribute dimensions of all data to be fused.

6. The method for fusing and consistently characterizing the multi-source heterogeneous data single frame applied to the unmanned system according to claim 1, wherein the dimension transformation relation in step four refers to a data transformation method between an original data set and a new data set when the original data is changed in data dimension, the data transformation method can change the form and description mode of the original data, but should maintain the integrity of data information as much as possible, and the data transformation method includes any one of grid generation, principal component analysis, factor analysis, linear combination and clustering.

7. The method according to claim 1, wherein the confidence discrimination processing of step four is a processing strategy for the occurrence of a conflict at the same data space point after the multi-source heterogeneous data is converted into the same data description, and the confidence parameters include a priori-based data source priority, raw data error statistics in preprocessing, data density, and data continuity, and finally the data after conflict merging is obtained through weighted calculation according to the confidence parameters.

Images