CN118429764A - Collaborative sensing method based on multi-mode fusion - Google Patents

Abstract

The invention provides a collaborative sensing method based on multi-mode fusion, which adopts a method for integrating laser radar and camera data, and by fusing the data of two sensors into the same representation space, the accurate depth information provided by the laser radar and the rich visual features provided by the camera can be utilized to realize more accurate and more stable environment sensing, and a plurality of different fusion strategies are considered and are deeply analyzed. The weight of different sensor data can be adaptively adjusted, so that the correlation between the sensor data and the sensor data can be captured better, the high-precision sensing of the environment is realized, and the sensor data display method has better performance and stronger robustness in the cooperative sensing task.

Description

Collaborative perception method based on multimodal fusion

Technical Field

The present invention relates to the field of autonomous driving and intelligent transportation systems, and more specifically to a collaborative perception method of multimodal fusion.

Background technique

Collaborative perception is a key issue in the field of autonomous driving. It allows autonomous vehicles to collect environmental information through on-board sensors and share it with other vehicles in real time through vehicle-to-vehicle (V2V) wireless communication technology, thereby achieving more powerful and comprehensive environmental perception capabilities. The goal of collaborative perception tasks is to use multiple vehicles as

Build high-precision multi-point observation scene representation for mobile sensor networks to enhance vehicle perception in application scenarios such as fleet coordination, traffic flow optimization, autonomous driving assistance and autonomous driving. Compared with traditional single-vehicle intelligence, multi-vehicle collaborative perception can better cope with complex traffic scenarios and improve the decision-making accuracy and driving safety of autonomous driving systems.

Early research was mainly based on a single sensor modality, such as using only LiDAR or only cameras for environmental perception. However, these single-modality approaches fail to fully exploit the complementary advantages of both sensors, limiting the performance of the perception system. Pure LiDAR-based methods may ignore the fine-grained visual details captured by camera sensors, while pure camera-based methods usually lack accurate depth information that is critical for precise object localization and may also be affected by low light or bad weather conditions.

In recent years, researchers have begun to explore perception methods based on multimodal fusion in order to overcome the limitations of single modality. These methods aim to combine the advantages of lidar and cameras to obtain more comprehensive and accurate environmental perception, and have demonstrated the advantages of fusing lidar and camera data, including improved target detection, enhanced scene understanding, and robust performance under challenging environmental conditions. However, when it comes to multi-vehicle collaborative perception tasks, the exploration of multimodal fusion is still very limited. The recently proposed HM-ViT is an early exploration of multimodal fusion in multi-vehicle collaborative perception. It assumes that each vehicle can only obtain any one modality, and based on this setting, heterogeneous modal fusion between different vehicles is achieved. Although there are a lot of studies on multimodality in the existing literature, in-depth discussions on multimodal interaction are still insufficient in the field of collaborative perception.

Summary of the invention

In order to overcome the shortcomings of the prior art, the present invention provides a collaborative perception method based on multimodal fusion. In order to more comprehensively utilize the complementary advantages of different types of sensors, the present invention constructs a baseline system of multimodal fusion as the basic framework for realizing collaborative perception. The baseline system refers to the successful case of multimodal fusion in single-vehicle perception, and adopts a method of integrating lidar and camera data. By fusing the data of the two sensors into the same representation space, it can utilize the precise depth information provided by the lidar and the rich visual features provided by the camera to achieve more accurate and robust environmental perception. When constructing the baseline system, the present invention considers a variety of different fusion strategies and conducts in-depth analysis. First, the channel-level splicing and element-level summation methods are adopted. These two methods are simple and intuitive, and can directly splice features from different sensors together to form a unified feature representation. Although this method can achieve good performance in some cases, it also has certain limitations, mainly manifested in that it cannot fully consider the correlation between different sensor data. In order to solve this problem, the present invention further explores a fusion method based on the attention mechanism, which can adaptively adjust the weights of different sensor data to better capture the correlation between them. Through the multi-head self-attention mechanism, the proposed algorithm can establish complex associations between different sensor features, thereby achieving more sophisticated fusion. The collaborative perception method proposed in the present invention describes how to use two different sensors, lidar and camera, to achieve high-precision perception of the environment through advanced data processing and fusion technology. This method is not limited to a specific system, but can be used in different systems. The experimental results on the OPV2V dataset show that this fusion method based on the attention mechanism shows superior performance and stronger robustness in collaborative perception tasks compared with traditional fusion methods.

The steps of the technical solution adopted by the present invention to solve its technical problem are as follows:

Step 1: Multimodal feature extraction: For multi-view image data, the CaDDN (Categorical Depth Distribution Network) architecture is used. The CaDDN architecture includes four modules: encoder, depth estimation, voxel transformation, and folding. It ensures that as rich and accurate information as possible is captured from the input image, so that the extracted image features can fully represent the key visual information of the input image, including edge, texture, color, and scene depth. For point cloud data, PointPillar is selected as the feature extractor of point cloud data. First, for a given 3D point Determine the position p = (i, j, l) of point p in the cylindrical coordinate system; divide the three-dimensional space of the 3D point p into a series of evenly spaced cylindrical structures, converting the complex 3D data into a 2D structure; then, all the cylindrical structures are flattened along the height direction of the cylindrical structure itself to generate a pseudo image; finally, further feature encoding and integration are performed through a series of two-dimensional convolutions;

The two-dimensional convolutional network is used to extract spatial features and strengthen the feature correlation between different columns. Through layer-by-layer convolution and activation functions, the network can capture and encode more complex spatial patterns, thereby improving the expressiveness of features. After being processed by the convolutional network, a set of encoded high-dimensional feature maps is obtained. The high-dimensional feature map represents the rich spatial information and object features in the original point cloud data. The high-dimensional feature map is integrated through pooling and splicing operations to form the final feature representation for subsequent detection tasks.

Step 2: Multimodal feature fusion: Each vehicle first compresses and encodes its own BEV features and then sends them to the central vehicle. When the central vehicle successfully receives the BEV features from all other vehicles, it fuses the received BEV features to generate a more global and detailed scene representation. The fusion process will first perform homogeneous modal feature fusion and then heterogeneous modal feature fusion.

Step 3: Detection head network: The detection head network predicts the category and position of the target based on the fused features. The detection head network contains 3 deconvolution layers. The 3 deconvolution layers are responsible for upsampling the feature maps to provide more fine-grained information for subsequent category and position predictions. After being processed by the 3 deconvolution layers, the feature maps are upsampled and concatenated together to form the category prediction branch and the bounding box regression branch. The category prediction branch outputs a score for each anchor box, indicating the vehicle category in the anchor box and the probability of its existence. The bounding box regression branch is responsible for refining the location information of the target and predicting 4 values for each anchor box, representing the offset of the center position (x, y) and the changes in the width and height of the box. Combining the outputs of the category prediction branch and the bounding box regression branch, the detection head network outputs the target category and adjusted location information in each anchor box.

The steps of isomorphic modality feature fusion are as follows:

For the image BEV features, the element-level maximization operation is used to fuse the image BEV features of different vehicles. By comparing the image BEV features of each vehicle and taking the maximum value at the element level, it is ensured that the final fused features contain the most significant features observed by all vehicles; specifically, when the central vehicle receives the image BEV features sent by other vehicles When , it is element-by-element compared with its own image BEV feature Perform element-by-element comparison and then select the maximum value as the fusion result; this process is expressed as

The max(·) operation is performed element by element, which ensures that the fused features inherit the most significant and rich parts of multiple vehicles;

For the point cloud BEV features, the self-attention mechanism is used for fusion. This process requires the establishment of a locality graph, which is a data structure used to represent the relationship between feature vectors of the same spatial position in different vehicles. In actual operation, the BEV feature vector of each vehicle is first determined and mapped to a unified coordinate system. Then, for each BEV feature vector, a node is created in the local map and connections are established between spatially adjacent nodes. For each node (i.e., feature vector) in the local map, the attention score between it and the adjacent nodes is calculated. The attention score determines the degree of influence of the information of each adjacent node on the current node during the fusion process. Then, according to the calculated attention score, the feature vector of each node is updated to the weighted sum of the features of its neighboring nodes. After iterative updates by the self-attention mechanism, the feature vector of each node fuses information from different vehicles but in the same or similar spatial positions. Finally, the updated feature vectors are aggregated to form a fused BEV feature representation for subsequent detection tasks.

The steps of heterogeneous modality feature fusion are as follows:

In the heterogeneous modality fusion stage, the fused BEV features of different modalities are integrated to utilize the complementary information between the lidar and camera data. As shown in Figure 2, the present invention adopts three common fusion strategies: channel-level splicing, element-level summation, and Transformer fusion; when it is necessary to use all the feature information of image and point cloud data at the same time, channel-level splicing fusion is adopted. This method is simple and intuitive, and is suitable for situations where the features are directly correlated and the dimensions are matched; when it is necessary to quickly and easily combine the features of the two data sources, element-level summation fusion is adopted, which is suitable for situations where the feature dimensions are the same and the physical meanings corresponding to each element are similar; when the environment or scene being processed is highly complex and it is necessary to deeply explore the potential and non-intuitive associations between the lidar data and the image data, Transformer fusion is adopted.

The steps of channel-level splicing and fusion are:

First, the image and point cloud features are concatenated along the channel dimension to obtain the feature tensor, and then the concatenated features are sent to two convolutional layers for further processing; the first convolutional layer is used to capture spatial relationships and extract valuable features from the concatenated data; then, the second convolutional layer ensures that key information is retained after information compression, and reduces the channel dimension to further refine the fusion features. The two convolutional layers do not have to be universal and can be designed separately according to the characteristics of the features;

The steps of element-level summation and fusion are:

The lidar features are fed into a 1×1 convolutional layer to downsample the channel dimension. Then, the image features and the downsampled lidar features are added element by element to obtain the fused features.

The steps of Transformer fusion are:

First, the BEV features of the image and point cloud are connected along the channel dimension to form a joint feature tensor, and then the position encoding is combined with the feature vector by addition to increase the model's perception of the spatial position of the feature; then, in order to capture the complex interaction between the two modalities, a multi-head self-attention mechanism is introduced after adding the position encoding to the joint feature tensor, assigning different weights to each part of the different modalities, and enabling the model to capture the correlation between different modal features from multiple angles; then, the nonlinear processing ability of the model is further enhanced through the feedforward network, and finally, residual connection and layer normalization techniques are used in each step of Transformer fusion to obtain the final fusion features.

The beneficial effect of the present invention lies in that it deeply explores the applicability and effectiveness of several advanced fusion strategies in multimodal collaborative perception tasks, introduces a fusion method based on the attention mechanism, and performs well in improving the accuracy and robustness of the perception system; the present invention establishes complex associations between different sensor features through a multi-head self-attention mechanism, improves the fineness of fusion, demonstrates excellent performance and robustness, and verifies the effectiveness of multimodal fusion, providing an economically feasible solution for practical applications in the real world.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a framework diagram of the present invention;

Figure 2 shows three strategies for heterogeneous modal feature fusion. Figure 2 (a) shows the channel-level concatenation fusion strategy, Figure 2 (b) shows the element-level summation fusion strategy, and Figure 2 (c) shows the Transformer fusion strategy.

Figure 3 is a detailed diagram of the model architecture;

Figure 4 is a comparison of the impact of positioning error on model performance. Figure 4 (a) is a comparison of three modes of CARLA default town on the AP@0.5 indicator, Figure 4 (b) is a comparison of three modes of CARLA default town on the AP@0.7 indicator, Figure 4 (c) is a comparison of three modes of Culver City digital town on the P@0.5 indicator, and Figure 4 (d) is a comparison of three modes of Culver City digital town on the P@0.7 indicator.

Figure 5 is a visual comparison of the detection results of different models. Figure 5(a) is the pure image mode, Figure 5(b) is the pure point cloud mode, and Figure 5(c) is the Transformer fusion mode.

Detailed ways

The present invention is further described below in conjunction with the accompanying drawings and embodiments.

The present invention provides a collaborative perception method based on multimodal fusion, as shown in Figure 1, which is the core component of the framework, covering multimodal feature extraction, multimodal feature fusion and detection head network. Each component works together to achieve effective integration of lidar and camera data, fully tapping the complementarity of the two to improve perception accuracy in V2V scenarios. The specific steps of the method include:

Step 1: Multimodal feature extraction

To capture and preserve unique cues from different modalities, we use separate branches for feature extraction and generate a unified BEV representation.

(1) Multi-view image data

The present invention adopts the CaDDN (Categorical Depth Distribution Network) architecture, which includes four main modules: encoder, depth estimation, voxel transformation and folding, to ensure that as rich and accurate information as possible is captured from the input image, so that the extracted image features can fully represent the key visual information of the input image, such as edges, textures, colors, and scene depth. In order to fuse the two heterogeneous features of 2D image features and 3D point cloud features, it is necessary to explicitly predict the depth of each pixel in the image features to promote the 2D plane to 3D space, and finally convert it to a unified BEV space. Taking vehicle a _i as an example, first, the encoder module processes the original input image Perform preliminary feature extraction to generate image features with dimensions of X×Y×D The process is expressed as:

Then, the depth estimation module predicts a depth probability distribution P for each pixel, expressed as:

This distribution reflects the depth information of the pixel. Afterwards, the voxel conversion module projects the previously extracted features from 2D to 3D space. It generates the corresponding 3D voxel features based on all possible depth distributions and the image calibration matrix.

Finally, the folding module merges the 3D voxel features into a height plane.

in For the generated BEV feature map, H and W represent the height and width of the image BEV grid, and C represents the number of channels. This process can effectively obtain semantically rich visual features.

(2) Point cloud data

Due to the unique sparsity and three-dimensional structure of point cloud data, directly using traditional 3D convolutional networks is likely to cause huge computational and memory overhead. In order to effectively and efficiently extract features from such data, this paper uses PointPillar as a feature extractor for point cloud data. First, for a given 3D point The position of a point p in a cylindrical coordinate system can be defined as p = (i, j, l). Here, i and j are the x and y coordinates of the two-dimensional grid, respectively, and l represents its height in the vertical direction. The three-dimensional space that determines the 3D point p is then divided into a series of evenly spaced cylindrical structures. Formally, this division is expressed as

Where W _pillar and H _pillar represent the width and height of the pillar in the x and y directions respectively. In this way, complex 3D data can be converted into a 2D structure, and each point in the pillar shares the same height information. Subsequently, all pillars are flattened along the height direction of the pillar itself to generate a pseudo image. For all 3D points in the pillar, the corresponding 3D features are (where N _p is the number of points in the cylinder and C _p is the dimension of the point cloud feature) is converted into a 2D cylinder feature

Among them, φ(·) is a conversion function used to convert all points in the pillar into the overall representation of the pillar. Finally, a series of two-dimensional convolutions are used to further encode and integrate the features of F _pillar to obtain the BEV feature with dimensions of H×W×C Its dimension is the same as the image BEV feature. In this step, a two-dimensional convolutional network is used to extract spatial features and strengthen the feature correlation between different columns. Through layer-by-layer convolution and activation functions, the network can capture and encode more complex spatial patterns, thereby improving the expressiveness of features. After processing by the convolutional network, a set of encoded high-dimensional feature maps can be obtained. These feature maps represent the rich spatial information and object features in the original point cloud data. Finally, these high-dimensional feature maps will be integrated through operations such as pooling and splicing to form the final feature representation for subsequent detection tasks.

Step 2: Multimodal feature fusion

With the acquisition of BEV features extracted from lidar and cameras, the feature fusion link becomes critical. In this stage, each vehicle first compresses and encodes its own BEV features and then sends them to the central vehicle. When the central vehicle successfully receives the BEV features from all other vehicles, it fuses these received BEV features to generate a more global and detailed scene representation. The fusion process mainly involves two key steps: first, homogeneous modal feature fusion, and then heterogeneous modal feature fusion. Among them, homogeneous feature fusion mainly processes information from the same sensor type, while heterogeneous feature fusion aims to combine information from different types of sensors to fully explore the complementarity between different modalities.

(1) Homogeneous modal feature fusion

In the feature fusion between multiple modalities, the fusion of homogeneous features is obviously more intuitive and simpler than that of heterogeneous features, because they come from the same data source and share similar characteristics and distribution. In view of this feature, the present invention designs different fusion strategies for image BEV features and point cloud BEV features respectively, so as to integrate multi-source information under the same modality in a more direct way.

First, the present invention adopts element-level maximization operation to fuse the image BEV features of different vehicles. The principle behind it is that in a multi-vehicle collaborative environment, some features that may be observed by some vehicles may be more obvious or clearer than those of other vehicles. By comparing the image BEV features of each vehicle and taking the maximum value at the element level, it can be ensured that the final fused features contain the most significant features observed by all vehicles. Specifically, when the center vehicle receives the image BEV features sent by other vehicles When , it will compare it with its own image BEV feature element by element. Perform element-by-element comparison and then select the maximum value as the fusion result. This process can be expressed as

The max(·) operation is performed element by element, which ensures that the fused features inherit the most significant and rich parts of multiple vehicles.

For point cloud BEV features, the present invention uses a self-attention mechanism for fusion. The fusion of point cloud BEV features first involves the construction of a local map. This map connects feature vectors from different vehicles but at the same spatial position. In this process, each feature vector is regarded as a node, and the edge between two feature vectors is used to connect feature vectors at the same spatial position from different vehicles. Such a connection not only helps to fuse information from different data sources, but also provides a basis for subsequent self-attention mechanism processing.

Then, the self-attention mechanism is applied to the local map to perform point cloud BEV feature fusion. The core of the self-attention mechanism is that it can assign different weights to different feature vectors, which reflect the interdependence between features. In the context of point cloud BEV features, this means that each feature point is not only defined by its own attributes, but also affected by the surrounding feature points. Specifically, for each vehicle’s point cloud BEV feature The two-dimensional feature map is expanded into a one-dimensional feature vector of dimension M×C, where M=H×W. Then, the feature vector is converted into three parts: query (Q), key (K), and value (V), and then the updated feature vector is calculated through the self-attention formula This process can be described as

Among them, d _k is the dimension of the key vector, and the softmax(·) function ensures that all weights add up to 1. Finally, the updated feature vector is transformed to the original dimension H×W×C. The BEV features of all vehicles are updated in the above way and stacked to obtain the fusion result

By fusing multi-source homogeneous modal features, the resulting fusion features better integrate and express observation information from the same sensors of multiple vehicles, capturing the richness and diversity of environmental details from multiple angles. At the same time, this also lays a solid foundation for subsequent heterogeneous feature fusion. Deep fusion of complementary information between different sensing modalities can further enrich environmental descriptions and give full play to the value of various sensing resources.

(2) Heterogeneous modal feature fusion

In the heterogeneous modality fusion stage, the present invention integrates the fused BEV features of different modalities to utilize the complementary information between the lidar and camera data. As shown in Figure 2, the present invention adopts three common fusion strategies: channel-level splicing, element-level summation, and Transformer fusion to effectively combine modalities and generate a comprehensive representation. When it is necessary to utilize all the feature information of image and point cloud data at the same time, channel-level splicing fusion is adopted. This method is simple and intuitive, and is suitable for situations where the features are directly correlated and the dimensions are matched. When it is necessary to quickly and easily combine the features of two data sources, element-level summation fusion is adopted, which is suitable for situations where the feature dimensions are the same and the physical meanings corresponding to each element are similar. Transformer fusion is particularly applicable when the environment or scene being processed is highly complex and it is necessary to deeply explore the potential and non-intuitive associations between lidar data and image data. Specifically, given the fused image features and point cloud features The fusion process is as follows:

① Channel-level splicing and fusion

Channel-level splicing is an intuitive and widely used feature fusion method. This method mainly relies on the spatial consistency of features, that is, features from different modalities at the same spatial position are considered to be related. In channel-level splicing fusion, the image and point cloud features are first spliced along the channel dimension to obtain a feature tensor B ^cat with a dimension of H×W×4C. The spliced features are then fed into two convolutional layers for further processing. Among them, the first convolutional layer is used to capture spatial relationships and extract valuable features from the spliced data. Subsequently, the second convolutional layer needs to ensure that key information is retained after information compression, and the channel dimension is reduced to further refine the fused features. These two convolutional layers do not have to be universal and can be designed separately according to the characteristics of the features.

②Element-level summation and fusion

The element-wise summation aims to fuse the complementary information of the two modalities while maintaining the spatial structure. In the element-wise summation fusion module, the lidar features are first A 1×1 convolutional layer is used to downsample the channel dimension. Then, the image features and the downsampled lidar features are added element by element to obtain the fused features. This element-wise and summation feature fusion strategy ensures that the two modalities contribute equally to the final fusion result.

③Transformer Fusion

In recent years, the Transformer architecture has achieved great success in the field of natural language processing, especially in capturing long-distance dependencies in sequence data. In multimodal fusion, Transformer has attracted widespread attention due to its excellent performance and ability to handle long-distance dependencies. In particular, its internal self-attention mechanism provides a powerful framework for the interaction and interaction between different modal features. In the process of heterogeneous modal feature fusion, the features of each modality can be regarded as a "sequence", in which each "element" represents a part of the spatial information.

First, the BEV features of the image and point cloud are concatenated along the channel dimension to form a joint feature tensor

Considering that the original Transformer structure does not contain any information about the position of the element, it is necessary to combine the position encoding into the feature vector by addition. To this end, the position encoding is added to the joint feature to ensure that the model can recognize the position of the feature in space.

Next, in order to capture the complex interaction between the two modalities, the present invention introduces a multi-head self-attention mechanism. This mechanism enables the model to assign different weights to each part of different modalities. The multi-head design enables the model to capture the correlation between different modal features from multiple angles.

Then, the nonlinear processing capability of the model is further enhanced through the feedforward network.

B ^FFN =FeedForwarNetwork(B ^att )

Finally, to ensure the stability of the model and maintain the feature scale, residual connection and layer normalization techniques are used in each step of the above Transformer fusion to obtain the final fusion feature.

Overall, this Transformer-based fusion strategy allows the algorithm to fully consider and utilize the complex interactions and dependencies between the two modalities, thereby providing a highly rich and representative feature representation for downstream tasks.

Step 3: Detection Head Network

The detection head network is responsible for predicting the category and location of the target based on the fused features. The network contains three deconvolution layers, which are responsible for upsampling the feature maps to provide more fine-grained information for subsequent category and location predictions. After being processed by the three deconvolution layers, the feature maps are upsampled and concatenated together to form two branches in the detection head: the category prediction branch and the bounding box regression branch. The category prediction branch outputs a score for each anchor box, indicating whether there is an object of a certain vehicle category in the anchor box, and the probability of existence. These output scores can be converted into probability distributions of each category after being processed by the activation function. On the other hand, the bounding box regression branch is responsible for refining the location information of the target. It predicts 4 values for each anchor box, which represent the offset of the center position (x, y) and the change in the width and height of the box. Through these predicted values, the position of the anchor box can be corrected to make it surround the target object more closely. Combining the outputs of the two branches, the detection head network outputs the target category and adjusted location information in each anchor box.

The effects of the present invention are further described below in conjunction with experiments.

1. Experimental setup:

The experiment of the present invention was conducted on the OPV2V dataset. The dataset has a total of 11,464 frames, which are mainly divided into two subsets: the CARLA default town subset and the Culver City digital town subset. Among them, the CARLA default town subset accounts for 10,914 frames, which is divided into three parts: training, verification and testing according to the ratio of 6,764 frames, 1,980 frames and 2,170 frames. This subset covers a variety of scenes of different complexity, providing sufficient training and evaluation data for the collaborative perception model. Relatively speaking, the CulverCity subset has only 550 frames, but its goal is to evaluate the generalization performance of the model in real-world scenarios, especially those actual urban environments that pose challenges to the model's perception capabilities.

In terms of implementation details, the proposed algorithm is based on the PyTorch framework and trained on a PC with an NVIDIA RTX4090 GPU equipped with 24GB RAM. During the training process, a group of vehicles that can establish communication in the scene are randomly selected, and the communication range of each vehicle is set to 70m. At the same time, the range of the point cloud is set to [-140.8, 14.8] × [-40, 40] × [-3, 1] along the x, y and z axes. The resolution of the voxel is 0.4m. In order to improve the diversity of training data, a series of data enhancement techniques are used, such as random flipping, scaling within ±0.05, and rotation within ±45°. The model is trained using the Adam optimizer, and the initial learning rate of the model is set to 0.002 and the batch size is 2. In addition, an early stopping strategy based on validation loss is used to prevent the model from overfitting. The detailed architecture and other parameters of the algorithm of the present invention are shown in Figure 3.

In terms of performance evaluation indicators, the present invention uses a standard evaluation indicator to evaluate the performance of the algorithm, namely, Average Precision (AP). Average Precision is a commonly used target detection performance indicator, which is used to measure the accuracy of the algorithm under different Intersection over Union (IoU) thresholds. When calculating the average precision, it is necessary to set the threshold of the intersection over Union (IoU) in advance, that is, the ratio of the overlapping area of the prediction box and the true value box to the total area of the prediction box and the true value box. If it is greater than the set threshold, the detection is considered correct. The present invention calculates the AP value of the model when the IoU threshold is 0.5 and 0.7, namely AP@0.5 and AP@0.7. These two indicators can provide performance evaluation of the algorithm under different degrees of strictness, thereby more comprehensively measuring the performance of the algorithm in the target detection task.

According to different heterogeneous modal fusion strategies, the present invention constructs three versions of multimodal fusion models: channel-level splicing fusion, element-level summation fusion, and Transformer fusion. In addition, in order to more comprehensively verify the effect of the proposed multimodal fusion model, the algorithm of the present invention is also compared with a variety of existing advanced algorithms, including Cooper, F-Cooper, V2VNet, AttFuse and CoBEVT.

In addition to the general scenario where all vehicles can obtain data in both point cloud and image modes, the present invention also considers a more realistic and complex heterogeneous modality scenario. In this heterogeneous modality scenario, each vehicle can only obtain one of the modalities of point cloud and image, which is used to simulate the performance of the proposed algorithm in the absence of various modalities.

2. Analysis of quantitative experimental results:

In Table 1, the present invention comprehensively compares the performance of various SOTA algorithms on the OPV2V dataset.

Table 1 Comprehensive performance comparison with SOTA algorithm

First, all collaborative perception algorithms are significantly better than single-vehicle perception in different indicators in different scenarios, which shows that the mutual cooperation between multiple vehicles is conducive to better perception of the surrounding environment. Secondly, it can be found that the early fusion strategy performs better than the late fusion in most cases. This shows that without considering the communication bandwidth, the fusion of data in the early stage can better preserve the original scene information. Moreover, the performance of the three implementation versions of the algorithm of the present invention using different fusion strategies (i.e., element-level summation fusion, channel-level splicing fusion and Transformer fusion) is highly competitive among all the comparison algorithms, especially the Transformer fusion, a multimodal fusion model using the Transformer architecture, which achieves higher AP scores under both IoU thresholds. Although the Transformer fusion is slightly lower than the CoBEVT algorithm in the AP@0.7 indicator in the CARLA default town subset, it is ahead of all other comparison algorithms in both AP@0.5 and AP@0.7 indicators in the more challenging Culver City digital town subset.

Table 2 shows the performance comparison of the proposed algorithm in heterogeneous modal scenarios.

Table 2 Performance comparison of the proposed algorithm in different heterogeneous modal scenarios

Among them, the center vehicle image modality means that half of all vehicles can only obtain image data (including the center vehicle) and the other half can only obtain point cloud data; in contrast, the center vehicle point cloud modality means that half of the number of vehicles (including the center vehicle) can only obtain point cloud data. As can be seen from Table 2, the performance in the pure image mode is significantly lower, especially in the Culver City scene, AP@0.7, which is only 8.6%. The performance of the pure point cloud mode is significantly better than the pure image mode, mainly because 3D point clouds can provide accurate scene depth measurement, so they are more suitable for 3D object detection tasks. Further considering the mixed situation of heterogeneous modalities, both the center vehicle image modality and the center vehicle point cloud modality show relatively good performance. Especially in the case where the center vehicle can only obtain point cloud data (center vehicle point cloud modality), its performance is close to the pure point cloud mode, indicating that the data modality of the center vehicle plays a key role in V2V collaborative detection. For the center vehicle image modality, although its performance is slightly lower than that of the center vehicle point cloud modality, it is still significantly better than the pure image mode, indicating that even in the case where half of the vehicles can only obtain image data, the presence of point cloud data can still greatly enhance the detection performance of the system. The experimental results in Table 2 verify that the proposed algorithm has relatively robust performance in different heterogeneous modal scenarios, especially when modal data is lost. In addition, this also highlights the importance of point cloud data in V2V collaborative detection tasks and the impact of the center vehicle on the overall detection performance.

Positioning error is one of the complex factors that need to be considered in real scenes. In order to analyze the robustness of the model to positioning error, the present invention samples coordinate noise and angle noise from Gaussian distribution respectively, and adds them to accurate positioning data to simulate positioning error. Figure 4 shows the performance comparison of the proposed multimodal fusion algorithm (Transformer fusion) and the other two single-modal schemes (pure point cloud mode, pure image mode) under different positioning errors. It can be seen that with the increase of positioning noise, the performance of the three models tends to decline as a whole, especially the pure point cloud mode is most sensitive to positioning error: when the error increases to 0.2, the performance decreases by more than 4%; when the error is 0.4, it decreases by more than 10%. In contrast, the pure image mode is less affected by positioning error and the performance decreases slowly. The main reason is that the three-dimensional geometric features extracted from lidar data are more sensitive to the offset of the coordinate axis than the visual features extracted from the image. In addition, since the multimodal fusion model (Transformer fusion) contains certain redundant information so that it will not rely too much on a single mode, it has a strong tolerance for positioning error while maintaining high performance.

2. Analysis of qualitative test results:

Figure 5 shows the visualization detection results of three different schemes on the OPV2V dataset. As shown in the figure, the pure camera and pure lidar models are affected to varying degrees by factors such as semantic ambiguity and uncertainty, resulting in problems such as missed detection and false detection. In contrast, the multi-sensor fusion model proposed in this paper has achieved significant improvements in detection accuracy and robustness. By leveraging the complementary advantages of lidar and camera modes, the fusion model effectively addresses the limitations of a single sensor and achieves more accurate and reliable object detection.

Claims (6)

1. A collaborative perception method based on multimodal fusion, characterized by comprising the following steps: Step 1: Multimodal feature extraction: For multi-view image data, the CaDDN architecture is used. The CaDDN architecture includes four modules: encoder, depth estimation, voxel transformation, and folding. It ensures that as much rich and accurate information as possible is captured from the input image, so that the extracted image features can fully represent the key visual information of the input image, including edge, texture, color, and scene depth. For point cloud data, PointPillar is selected as the feature extractor of point cloud data. First, for a given 3D point Determine the position p = (i, j, l) of point p in the cylindrical coordinate system; divide the three-dimensional space of the 3D point p into a series of evenly spaced cylindrical structures, converting the complex 3D data into a 2D structure; then, all the cylindrical structures are flattened along the height direction of the cylindrical structure itself to generate a pseudo image; finally, further feature encoding and integration are performed through a series of two-dimensional convolutions; The two-dimensional convolutional network is used to extract spatial features and strengthen the feature correlation between different columns. Through layer-by-layer convolution and activation functions, the network can capture and encode more complex spatial patterns, thereby improving the expressiveness of features. After being processed by the convolutional network, a set of encoded high-dimensional feature maps is obtained. The high-dimensional feature map represents the rich spatial information and object features in the original point cloud data. The high-dimensional feature map is integrated through pooling and splicing operations to form the final feature representation for subsequent detection tasks. Step 2: Multimodal feature fusion: Each vehicle first compresses and encodes its own BEV features and then sends them to the central vehicle. When the central vehicle successfully receives the BEV features from all other vehicles, it fuses the received BEV features to generate a more global and detailed scene representation. The fusion process will first perform homogeneous modal feature fusion and then heterogeneous modal feature fusion. Step 3: Detection head network: The detection head network predicts the category and position of the target based on the fused features. The detection head network contains 3 deconvolution layers. The 3 deconvolution layers are responsible for upsampling the feature maps to provide more fine-grained information for subsequent category and position predictions. After being processed by the 3 deconvolution layers, the feature maps are upsampled and concatenated together to form the category prediction branch and the bounding box regression branch. The category prediction branch outputs a score for each anchor box, indicating the vehicle category in the anchor box and the probability of its existence. The bounding box regression branch is responsible for refining the location information of the target and predicting 4 values for each anchor box, representing the offset of the center position (x, y) and the changes in the width and height of the box. Combining the outputs of the category prediction branch and the bounding box regression branch, the detection head network outputs the target category and adjusted location information in each anchor box. 2. The collaborative perception method based on multimodal fusion according to claim 1, characterized in that: The steps of isomorphic modality feature fusion are as follows: For the image BEV features, the element-level maximization operation is used to fuse the image BEV features of different vehicles. By comparing the image BEV features of each vehicle and taking the maximum value at the element level, it is ensured that the final fused features contain the most significant features observed by all vehicles; when the central vehicle receives the image BEV features sent by other vehicles When , it is element-by-element compared with its own image BEV feature Perform element-by-element comparison and then select the maximum value as the fusion result; this process is expressed as The max(·) operation is performed element by element, which ensures that the fused features inherit the most significant and rich parts of multiple vehicles; For the point cloud BEV features, the self-attention mechanism is used for fusion. This process requires the establishment of a local map, which is a data structure used to represent the relationship between feature vectors of the same spatial position in different vehicles. In actual operation, the BEV feature vector of each vehicle is first determined and mapped to a unified coordinate system. Then, for each BEV feature vector, a node is created in the local map, and connections are established between spatially adjacent nodes. For each node in the local map, the attention score between it and the adjacent nodes is calculated. The attention score determines the degree of influence of the information of each adjacent node on the current node during the fusion process. Then, based on the calculated attention score, the feature vector of each node is updated to the weighted sum of the features of its neighboring nodes. After iterative updates by the self-attention mechanism, the feature vector of each node fuses information from different vehicles but in the same or similar spatial positions. Finally, the updated feature vectors are aggregated to form a fused BEV feature representation for subsequent detection tasks. 3. The collaborative perception method based on multimodal fusion according to claim 1, characterized in that: The steps of heterogeneous modality feature fusion are as follows: In the heterogeneous modal fusion stage, the fused BEV features of different modalities are integrated to utilize the complementary information between the lidar and camera data, and three common fusion strategies are adopted: channel-level stitching, element-level summation and Transformer fusion; when it is necessary to utilize all the feature information of image and point cloud data at the same time, channel-level stitching fusion is adopted. This method is simple and intuitive, and is suitable for situations where the features are directly correlated and the dimensions are matched; when it is necessary to quickly and easily combine the features of the two data sources, element-level summation fusion is adopted, which is suitable for situations where the feature dimensions are the same and the physical meanings corresponding to each element are similar; when the environment or scene being processed is highly complex and it is necessary to deeply explore the potential and non-intuitive associations between the lidar data and the image data, Transformer fusion is adopted. 4. The collaborative perception method based on multimodal fusion according to claim 3 is characterized in that: The steps of channel-level splicing and fusion are: First, the image and point cloud features are concatenated along the channel dimension to obtain the feature tensor, and then the concatenated features are sent to two convolutional layers for further processing; among them, the first convolutional layer is used to capture spatial relationships and extract valuable features from the concatenated data; then, the second convolutional layer ensures that key information is retained after information compression, and reduces the channel dimension to further refine the fusion features. The two convolutional layers do not have to be universal and can be designed separately according to the characteristics of the features. 5. The collaborative perception method based on multimodal fusion according to claim 3 is characterized in that: The steps of element-level summation and fusion are: The lidar features are fed into a 1×1 convolutional layer to downsample the channel dimension. Then, the image features and the downsampled lidar features are added element by element to obtain the fused features. 6. The collaborative perception method based on multimodal fusion according to claim 3 is characterized in that: The steps of Transformer fusion are: First, the BEV features of the image and point cloud are connected along the channel dimension to form a joint feature tensor, and then the position encoding is combined with the feature vector by addition to increase the model's perception of the spatial position of the feature; then, in order to capture the complex interaction between the two modalities, a multi-head self-attention mechanism is introduced after adding the position encoding to the joint feature tensor, assigning different weights to each part of the different modalities, and enabling the model to capture the correlation between different modal features from multiple angles; then, the nonlinear processing ability of the model is further enhanced through the feedforward network, and finally, residual connection and layer normalization techniques are used in each step of Transformer fusion to obtain the final fusion features.