CN117011656A - Panoramic camera and laser radar fusion method for obstacle avoidance of unmanned boarding bridge - Google Patents

Abstract

The invention discloses a panoramic camera and laser radar fusion method for obstacle avoidance of an unmanned boarding bridge, which comprises the following steps: extracting features of edges, textures, colors and the like of an environment image of the panoramic camera, and representing the features as high-dimensional vectors; extracting features from distance data of the laser sensor, and representing the features as high-dimensional vectors; the method can effectively utilize the data characteristics of the panoramic camera and the laser sensor, and realize matching, calibration, segmentation, identification and positioning of the environmental image and the distance data, thereby improving the accuracy, the speed and the stability of the depth estimation of the panoramic camera. The invention can effectively utilize the data characteristics of the panoramic camera and the laser sensor, and realize the matching, calibration, segmentation, identification and positioning of the environmental image and the distance data, thereby improving the accuracy, the speed and the stability of the depth estimation of the panoramic camera.

Description

A panoramic camera and lidar fusion method for obstacle avoidance on unmanned boarding bridges

Technical field

The invention relates to the technical field of obstacle avoidance sensing for unmanned docking bridges, and specifically relates to a panoramic camera and lidar fusion method for obstacle avoidance on unmanned boarding bridges.

Background technique

The unmanned docking bridge is an intelligent device that can automatically dock with aircraft. It can achieve the advantages of unattended, efficient, safe, energy-saving and environmentally friendly. In order to realize the target sensing function of the unmanned docking bridge such as obstacle avoidance and personnel, it is necessary to have real-time, accurate and comprehensive perception of the surrounding environment in order to adjust the movement trajectory and speed in a timely manner to avoid collisions and injuries.

Currently, commonly used environment sensing sensors include cameras, laser radar, ultrasonic waves, millimeter waves, etc. Among them, cameras can provide rich image information, but depth information is inaccurate or missing; lidar can provide accurate distance information, but has low resolution or small field of view; ultrasonic waves and millimeter waves can provide short-range obstacle avoidance information. But the interference is large or the cost is high. Therefore, it is difficult for a single sensor to meet the environmental sensing needs of unmanned docking bridges.

With the rapid development of high-speed communication and artificial intelligence technology, human perception of real-world scenes is no longer limited to the use of small field of view (FoV) and low-dimensional scene detection devices. Panoramic imaging emerged as the next generation of innovative smart instruments for environmental sensing and measurement.

Existing Technology (Review on Panoramic Imaging and Its Applications in SceneUnderstanding, Shaohua Gao Kailun Yang Hao Shi Kaiwei Wang and Jian Bai1) introduces the principle of panoramic cameras and explains the functions of panoramic cameras in panoramic semantic image segmentation, panoramic depth estimation, and panoramic visual positioning. The important value and restrictive factors in such directions, looking forward to the future potential and research direction of panoramic imaging instruments, also pointed out that although panoramic imaging instruments meet the needs of large-field photography and imaging, they also need to have high resolution, no blind spots, and miniaturization. And multi-dimensional intelligent perception capabilities need to be combined with artificial intelligence methods to achieve a deeper understanding and comprehensive perception of the 360° real environment.

The existing technology (Rethinking Supervised Depth Estimation for 360° PanoramicImagery Lu He, Bing Jian, Yangming Wen, Haichao Zhu, Kelin Liu, Weiwei Feng, ShanLiu Tencent America, Palo Alto, USA) proposes an estimation from a single 360° panoramic image Deep approach considers this a complex task. Estimating depth maps from RGB panoramic images becomes more difficult because of the inherent scale ambiguity problem. To solve this problem, they proposed a method to normalize depth data based on the estimated camera height to alleviate the scale inconsistency problem in real depth maps. In addition to this, they also designed a multi-head planar guided depth network aiming to provide more geometric constraints for depth estimation. Experimental results clearly show that relative depth estimation is more accurate than absolute depth estimation. This model shows good performance on both Matterport3D and Stanford2D3D datasets. Its limitation is: the proposed method requires an estimate of the camera height, which may not be available or accurate in some cases. The proposed method relies on the assumption of planar structure in the scene, which may not hold in some cases. Due to its computational complexity, the proposed method may not be suitable for real-time applications.

The above panoramic vision sensor is a method for environmental perception. Through a properly designed panoramic camera solution, coverage of the 360-degree field of view and depth estimation can be achieved. This method has the advantages of wide field of view, low cost, and easy installation, but it also has the following shortcomings:

a. The depth estimation accuracy of panoramic cameras is low and is easily affected by factors such as lighting and texture;

b. Panoramic camera depth estimation requires a large amount of calculation and is difficult to achieve real-time performance;

c. The depth estimation results of panoramic cameras are unstable and prone to missed detections, false detections, etc.

Therefore, how to improve the accuracy, speed and stability of panoramic camera depth estimation is an urgent technical issue to be solved in the field of environmental perception for unmanned docking bridges.

Contents of the invention

In order to solve the defects and shortcomings of the existing technology; the purpose of the present invention is to provide a panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance; it can effectively utilize the data characteristics of the panoramic camera and the laser sensor to achieve Match, calibrate, segment, identify and position environmental images and distance data to improve the accuracy, speed and stability of panoramic camera depth estimation.

The present invention is realized through at least one of the following technical solutions.

A panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance, including the following steps:

(S1). The data preprocessing module preprocesses the data from panoramic cameras and laser sensors, including denoising, filtering, and correction;

(S2), the feature extraction module extracts features from the environment image of the panoramic camera, uses a convolutional neural network to extract the edge, texture, and color features of the image, and represents the features as high-dimensional vectors;

(S3), perform feature extraction on the distance data of the laser sensor, use point cloud processing methods to extract the shape, size, and direction features of the distance data, and represent the features as high-dimensional vectors;

(S4). The feature matching module matches the feature vectors of the panoramic camera and the laser sensor, and uses the similarity measurement method to calculate the feature vectors between different sensors;

(S5). The data fusion module fuses the matching feature vectors and uses the weighted average method to fuse the feature vectors of different sensors into a unified feature vector that reflects the image and distance information of the environment;

(S6). The image segmentation module segments the fused feature vectors and uses the clustering method to divide the fused feature vectors into different areas, each area corresponding to a target or background;

(S7). The target recognition module identifies the segmented areas, uses the classification method, and determines the category of the area based on the feature vector of the area;

(S8). The target positioning module locates the identified target and uses the regression method to calculate the parameters of the target based on the feature vector of the region.

Further, in step (S1), the distortion correction of the distorted image collected by the fisheye camera is as follows:

x'＝x(1+k1r ² +k2r ⁴ +k3r ⁶ )

y'＝y(1+k1r ² +k2r ⁴ +k3r ⁶ )

Among them, x and y are the coordinates on the distorted image, x' and y' are the coordinates on the corrected image, r is the distance from the point on the distorted image to the center point, k1, k2 and k3 are the distortion coefficients.

Further, the feature extraction of the RGB-D image in step (S2) is:

f＝W*x+b

Among them, f is the feature vector, W is the convolution kernel, x is the RGB-D image, and b is the bias term.

Further, in step (S3), the following formula is used to extract features from the point cloud data:

f＝g(h(p))

Among them, f is the feature vector, g is the nonlinear activation function, h is the point cloud processing function, and p is the point cloud data.

Further, in step (S4), the following formula is used to calculate the similarity:

s=cos(f1,f2)=f1*f2/(|f1|*|f2|)

Among them, s is the similarity, cos is the cosine function, f1 and f2 are the feature vectors of different sensors.

Further, in step (S5), the following formula is used for data fusion:

F＝a*f1+(1-a)*f2

Among them, F is the fused feature vector, a is the weight coefficient, f1 and f2 are the feature vectors of different sensors.

Further, in step (S6), the following formula is used for image segmentation:

L＝argmin(S(F))

Among them, L is the segmented label, S is the image segmentation function, and F is the fused feature vector.

A system that implements the panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance, including a panoramic camera module, a laser sensor module, a data preprocessing module, a feature extraction module, a feature matching module, and a data fusion module. Image segmentation module, target recognition module and target positioning module; the panoramic camera module and laser sensor module are connected with the data preprocessing module, and the data preprocessing module is connected with the feature extraction module, feature matching module, data fusion module, image segmentation module, The target identification module and the target positioning module are connected to each other.

Further, the panoramic camera module is a dual panoramic camera. The dual panoramic camera includes six 120-degree fisheye cameras for collecting environmental image data. Each fisheye camera collects a field of view of 120 degrees. Two fisheye cameras are spliced into a 360-degree panoramic camera, and two panoramic cameras form a binocular panoramic camera to achieve coverage of the 360-degree field of view and estimate depth.

Further, the laser sensor module includes a rotating multi-line lidar.

Compared with existing technology, the beneficial effects of the present invention are:

1. Able to effectively utilize the data characteristics of panoramic cameras and laser sensors to achieve matching, calibration, segmentation, identification and positioning of environmental images and distance data;

2. It can effectively improve the accuracy, speed and stability of panoramic camera depth estimation, and improve the performance of unmanned docking bridge obstacle avoidance and target sensing functions such as people;

3. It can effectively reduce the calculation amount and bandwidth requirements of data processing and transmission, and is suitable for embedded devices and wireless network environments.

Description of the drawings

In order to more clearly illustrate the embodiments of the present invention or technical solutions in the prior art, the present invention is described in detail with the following specific implementations and drawings.

Figure 1 is a structural diagram of a panoramic camera and lidar fusion system for unmanned boarding bridge obstacle avoidance according to an embodiment of the present invention;

Figure 2 is a schematic diagram of a panoramic camera module according to an embodiment of the present invention;

Figure 3 is a step flow chart of a panoramic camera and lidar fusion system for unmanned boarding bridge obstacle avoidance according to an embodiment of the present invention;

Explanation of reference signs: hardware system 1, software system 2;

Panoramic camera module 11, laser sensor module 12;

Data preprocessing module 21, feature extraction module 22, feature matching module 23, data fusion module 24, image segmentation module 25, target recognition module 26, target positioning module 27.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is described below through the specific embodiments shown in the drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. Furthermore, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily confusing the concepts of the present invention.

Here, it should also be noted that, in order to avoid obscuring the present invention with unnecessary details, only the structures and/or processing steps closely related to the solution according to the present invention are shown in the drawings, and the details related to them are omitted. Other details are less relevant to the invention.

As shown in Figures 1-3, this panoramic camera and lidar fusion system for unmanned boarding bridge obstacle avoidance includes hardware system 1 and software system 2;

The hardware system 1 includes a panoramic camera module 11 and a laser sensor module 12;

The software system 2 includes a data preprocessing module 21 , a feature extraction module 22 , a feature matching module 23 , a data fusion module 24 , an image segmentation module 25 , a target recognition module 26 and a target positioning module 27 .

The panoramic camera module 11 and the laser sensor module 21 are both connected to the data preprocessing module 21, and the data preprocessing module 21 is connected to the feature extraction module 22, feature matching module 23, data fusion module 24, image segmentation module 25, and target recognition module 26 and the target positioning module 27 are connected to each other.

The panoramic camera module 11 is a dual panoramic camera. The dual panoramic cameras can be installed at the front and rear ends of the unmanned docking bridge, corresponding to the front and rear observation circles respectively.

As a preferred embodiment, the dual panoramic cameras are composed of six 120-degree fisheye cameras and are used to collect environmental image data. Among them, each fisheye camera can capture a 120-degree field of view, three fisheye cameras can be spliced into a 360-degree panoramic camera, and two panoramic cameras can form a binocular panoramic camera to achieve a 360-degree field of view. Cover and estimate depth.

The laser sensor module 12 consists of a rotating multi-line laser radar and is used to collect environmental distance data. The lidar can rotate in the horizontal direction and emit multiple laser beams in the vertical direction to form a scanning plane. This scanning plane can be parallel to, and at a certain distance from, the observation circle of the dual panoramic camera. This lidar can collect sparse point cloud data on the surface of objects in the environment.

The data preprocessing module 21 is used to preprocess data from panoramic cameras and laser sensors, including denoising, filtering, correction and other operations, to improve the quality and consistency of the data. Specifically, this module can perform distortion correction on distorted images collected by fisheye cameras to restore them to normal viewing angles; calibrate and splice images collected by multiple fisheye cameras to form a complete panoramic image ; Perform stereo matching and depth estimation on the front and rear panoramic images collected by the dual panoramic cameras to form a complete RGB-D image; perform denoising, filtering, transformation and other operations on the point cloud data collected by the lidar, Make it in the same coordinate system as the RGB-D image and align it with it.

The feature extraction module 22 is used to extract features from the data of the panoramic camera and the laser sensor, and represent the features as high-dimensional vectors.

As a preferred embodiment, this module can use convolutional neural network (CNN) or other deep learning methods to perform feature extraction on RGB-D images, extract the edges, texture, color and other features of the object surface in the image, and Represent features as high-dimensional vectors; this module can also use point cloud processing methods to extract features from point cloud data, extract the shape, size, direction and other features of the object surface in the point cloud, and represent the features as high-dimensional vectors .

The feature matching module 23 is used to match the feature vectors of the panoramic camera and the laser sensor and calculate the similarity. Specifically, this module can use similarity measurement methods, such as cosine similarity, Euclidean distance, etc., to calculate the difference between the feature vector of each pixel in the RGB-D image and the feature vector of the nearest neighbor point in the point cloud data. similarity, and use similarity as the basis for matching. This module can also use thresholds or other screening methods to eliminate matching pairs whose similarity is lower than a certain value to improve the accuracy of matching.

The data fusion module 24 is used to fuse matching feature vectors and represent the fused feature vectors as a unified feature vector. Specifically, this module can use weighted average or other data fusion methods to perform a weighted average of the feature vector of each pixel in the RGB-D image and the feature vector of its nearest neighbor point in the matching point cloud data to obtain a fused eigenvector. This module can also use normalization or other standardization methods to make the fused feature vectors have a certain range and distribution to facilitate subsequent processing.

The image segmentation module 25 is used to segment the fused feature vectors and represent the segmented areas as different labels. Specifically, this module can use clustering or other image segmentation methods to segment the fused feature vector into different regions, each region corresponding to a target or background. The module can also use edge detection or other boundary extraction methods to extract the boundaries of each region and represent the boundaries as a set of continuous or discrete points.

The target recognition module 26 is used to identify the segmented areas and determine the category of the areas. Specifically, this module can use classification or other target recognition methods to determine which type of target the area belongs to based on the feature vector of each area, such as people, cars, obstacles, etc. The module can also use tags or other marking methods to give each area a unique identifier and display it on the image.

The target positioning module 27 is used to position the identified target and calculate the position, attitude, speed and other parameters of the target. Specifically, this module can use regression or other target positioning methods to calculate the position, attitude, speed and other parameters of the target in space based on the feature vectors and boundary points of each target area. This module can also use arrows or other indication methods to display the target's movement direction and speed on the image.

As shown in Figure 3, a panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance includes the following steps:

S1. Preprocess data from panoramic cameras and laser sensors, including denoising, filtering, and correction to improve data quality and consistency;

Among them, when correcting the distortion of the distorted image collected by the fisheye camera, the following formula is used:

x'＝x(1+k1r ² +k2r ⁴ +k3r ⁶ )

y'＝y(1+k1r ² +k2r ⁴ +k3r ⁶ )

In the above formula, x and y are the coordinates on the distorted image, x' and y' are the coordinates on the corrected image, r is the distance from the point on the distorted image to the center point, k1, k2 and k3 are the distortion coefficients.

S2. Extract features from the environment image of the panoramic camera, use convolutional neural network (CNN) or other deep learning methods to extract features such as edges, texture, and color of the image, and represent the features as high-dimensional vectors;

When extracting features from RGB-D images, the following formula is used:

f＝W*x+b

In the above formula, f is the feature vector, W is the convolution kernel, x is the RGB-D image, and b is the bias term.

S3. Extract features from the distance data of the laser sensor, use point cloud processing methods to extract the shape, size, and direction features of the distance data, and represent the features as high-dimensional vectors;

When extracting features from point cloud data, the following formula is used:

f＝g(h(p))

In the above formula, f is the feature vector, g is the nonlinear activation function, h is the point cloud processing function, and p is the point cloud data.

S4. Match the feature vectors of the panoramic camera and the laser sensor, and use the similarity measurement method to calculate the feature vectors between different sensors;

When calculating similarity, the following formula is used:

s=cos(f1,f2)=f1*f2/(|f1|*|f2|)

In the above formula, s is the similarity, cos is the cosine function, f1 and f2 are the feature vectors of different sensors.

S5. Fusion of matching feature vectors, using weighted average or other data fusion methods, to fuse the feature vectors of different sensors into a unified feature vector that reflects the image and distance information of the environment;

Among them, when performing data fusion, the following formula is used:

F＝a*f1+(1-a)*f2

In the above formula, F is the fused feature vector, a is the weight coefficient, f1 and f2 are the feature vectors of different sensors.

S6. Segment the fused feature vector and use clustering or other image segmentation methods to divide the fused feature vector into different areas, each area corresponding to a target or background;

Among them, when performing image segmentation, the following formula is used:

L＝argmin(S(F))

In the above formula, L is the segmented label, S is the image segmentation function, and F is the fused feature vector.

S7. Identify the segmented areas, use classification or other target recognition methods, and determine the category of the area based on the feature vector of the area, such as people, vehicles, obstacles, etc.;

Among them, when performing target recognition, the following formula is used:

C＝softmax(W*f+b)

In the above formula, C is the target category, softmax is the normalized exponential function, W is the classification weight, f is the feature vector of the region, and b is the classification bias term.

S8. Position the identified target, use regression or other target positioning methods, and calculate the target's position, attitude, speed and other parameters based on the feature vector of the area;

Among them, when performing target positioning, the following formula is used:

P＝W*f+b

In the above formula, P is the target position, attitude, speed and other parameters, W is the regression weight, f is the feature vector of the area, and b is the regression bias term.

It is obvious to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as illustrative and non-restrictive from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is therefore intended that all claims falling within the claims All changes within the meaning and scope of equivalent elements are included in the present invention.

In addition, it should be understood that although this specification is described in terms of implementations, not each implementation only contains an independent technical solution. This description of the specification is only for the sake of clarity, and those skilled in the art should take the specification as a whole. , the technical solutions in each embodiment can also be appropriately combined to form other implementations that can be understood by those skilled in the art.

Claims (10)

1. A panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance, which is characterized by: including the following steps: (S1). The data preprocessing module preprocesses the data from panoramic cameras and laser sensors, including denoising, filtering, and correction; (S2), the feature extraction module extracts features from the environment image of the panoramic camera, uses a convolutional neural network to extract the edge, texture, and color features of the image, and represents the features as high-dimensional vectors; (S3), perform feature extraction on the distance data of the laser sensor, use point cloud processing methods to extract the shape, size, and direction features of the distance data, and represent the features as high-dimensional vectors; (S4). The feature matching module matches the feature vectors of the panoramic camera and the laser sensor, and uses the similarity measurement method to calculate the feature vectors between different sensors; (S5). The data fusion module fuses the matching feature vectors and uses the weighted average method to fuse the feature vectors of different sensors into a unified feature vector that reflects the image and distance information of the environment; (S6). The image segmentation module segments the fused feature vectors and uses the clustering method to divide the fused feature vectors into different areas, each area corresponding to a target or background; (S7). The target recognition module identifies the segmented areas, uses the classification method, and determines the category of the area based on the feature vector of the area; (S8). The target positioning module locates the identified target and uses the regression method to calculate the parameters of the target based on the feature vector of the region. 2. A panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance according to claim 1, characterized in that: in step (S1), the distortion correction of the distorted image collected by the fisheye camera is: : x'＝x(1+k1r ² +k2r ⁴ +k3r ⁶ ) y'＝y(1+k1r ² +k2r ⁴ +k3r ⁶ ) Among them, x and y are the coordinates on the distorted image, x' and y' are the coordinates on the corrected image, r is the distance from the point on the distorted image to the center point, k1, k2 and k3 are the distortion coefficients. 3. A panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance according to claim 1, characterized in that: in step (S2), the feature extraction of the RGB-D image is: f＝W*x+b Among them, f is the feature vector, W is the convolution kernel, x is the RGB-D image, and b is the bias term. 4. A panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance according to claim 1, characterized in that: in step (S3), the following formula is used to extract features from the point cloud data: f＝g(h(p)) Among them, f is the feature vector, g is the nonlinear activation function, h is the point cloud processing function, and p is the point cloud data. 5. A panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance according to claim 1, characterized in that: in step (S4), the following formula is used to calculate the similarity: s=cos(f1,f2)=f1*f2/(|f1|*|f2|) Among them, s is the similarity, cos is the cosine function, f1 and f2 are the feature vectors of different sensors. 6. A panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance according to claim 1, characterized in that: in step (S5), the following formula is used for data fusion: F＝a*f1+(1-a)*f2 Among them, F is the fused feature vector, a is the weight coefficient, f1 and f2 are the feature vectors of different sensors. 7. A panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance according to claim 1, characterized in that: in step (S6), the following formula is used for image segmentation: L＝argmin(S(F)) Among them, L is the segmented label, S is the image segmentation function, and F is the fused feature vector. 8. A system for implementing a panoramic camera and lidar fusion method for unmanned boarding bridge obstacle avoidance according to claim 1, characterized by: a panoramic camera module, a laser sensor module, a data preprocessing module, and a feature extraction module. , feature matching module, data fusion module, image segmentation module, target recognition module and target positioning module; the panoramic camera module and laser sensor module are connected to the data preprocessing module, and the data preprocessing module is connected to the feature extraction module and feature matching module , the data fusion module, the image segmentation module, the target recognition module and the target positioning module are connected to each other. 9. The panoramic camera and lidar fusion system for unmanned boarding bridge obstacle avoidance according to claim 8, characterized in that: the panoramic camera module is a dual panoramic camera, and the dual panoramic camera includes Six 120-degree fisheye cameras are used to collect environmental image data. Each fisheye camera collects a 120-degree field of view. Three fisheye cameras are spliced into a 360-degree panoramic camera. Two panoramic cameras form a Binocular panoramic camera covers 360-degree field of view and estimates depth. 10. The panoramic camera and lidar fusion system for unmanned boarding bridge obstacle avoidance according to claim 8, characterized in that the laser sensor module includes a rotating multi-line lidar.