CN115082885A - Point cloud target detection method, device, equipment and storage medium - Google Patents

Abstract

The application relates to the technical field of automatic driving, and provides a method, a device, equipment and a storage medium for detecting a point cloud target, wherein the method is used for acquiring local point cloud characteristics and global voxel characteristics; and fusing the local point cloud characteristics and the global voxel characteristics based on a preset double-encoder and decoder algorithm to generate a three-dimensional target frame and complete the detection of the point cloud target in the scene. According to the method, after the local point cloud characteristics and the global voxel characteristics are respectively obtained through the point cloud network and the three-position sparse convolution network, the three-dimensional target frame with multi-modal characteristics fused is generated through the preset double-encoder and fusion decoder algorithm, and therefore the detection of the point cloud target in the scene to be detected is completed. Therefore, the multi-mode fusion is adopted to determine the three-dimensional target frame, the precision and the efficiency of the point cloud target detection method are improved, and the technical problems of low precision and low efficiency of the existing point cloud target detection method are solved.

Description

Point cloud target detection method, device, equipment and storage medium

technical field

The present invention relates to the technical field of automatic driving, and in particular, to a method, device, device and storage medium for detecting a point cloud target.

Background technique

At present, with the rapid development of 3D sensor technologies such as depth cameras, structured light cameras, and LIDAR, 3D spatial data such as point clouds are becoming more and more easily obtained and applied to various real application scenarios, such as driverless cycling. , robotics and virtual reality. 3D object detection is a challenging problem in computer vision and benefits many downstream vision tasks. With the rapid development of Convolution Neual Networks (CNN), two-dimensional object detection has achieved great success. Compared with 2D images, 3D point clouds are disordered, sparse, and irregular, which makes it difficult for 2D detection methods to be directly applied to 3D scenes.

Existing 3D object detection methods can generally be divided into two categories: voxel-based point cloud object detection and raw point cloud-based object detection. The detection of point cloud objects based on voxel division first converts the sparse 3D point cloud into a compact spatial representation such as bird's-eye view or speed-up grid, and extracts the corresponding semantic features for each expression unit, and then efficiently converts the two The 3D convolutional neural network is migrated to the 3D space, and the 3D convolution algorithm is used to realize the extraction of voxel scene semantic information and 3D object detection for these compact spatial representations. These voxel-based methods are suitable for generating relatively accurate spatial 3D position proposals, however, these methods inevitably lose the geometric structure information used to achieve precise localization during the voxelization process. The 3D target detection algorithm based on the original point cloud directly takes the original point cloud and the corresponding features as input, and extracts the point semantic features rich in geometric information through the point cloud network constructed based on the Multilayer Perceptron (MLP). The Farthest Point Sampling (FPS) and the point set feature extraction module (Set Abstract, SA) are used to ensure the uniformity of spatial information extraction and the effectiveness of local perception. These methods can make up for the lack of information caused by voxelization and more effectively describe the geometric structure of three-dimensional space, but such methods often require more computational resources. Therefore, how to improve the low accuracy and low efficiency of the existing point cloud target detection methods has become an urgent technical problem to be solved.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide a point cloud target detection method, device, device and computer-readable storage medium, aiming to solve the technical problems of low precision and low efficiency of the existing point cloud target detection method.

In order to achieve the above object, the present invention provides a method for detecting a point cloud target. The method processes the three-dimensional voxel grid of the to-be-determined scene through a three-dimensional sparse convolutional neural network, and obtains the global voxel of the to-be-determined scene. feature, and generate a target proposal frame; process the original point cloud of the scene to be detected through the point cloud network to obtain the local point cloud features of the scene to be detected; based on the preset dual encoder and fusion decoder algorithm, fuse the local points The cloud feature and the global voxel feature are adjusted to the target proposal frame to generate a three-dimensional target frame, and based on the three-dimensional target frame, the point cloud target detection in the scene to be detected is completed.

In addition, in order to achieve the above purpose, the present invention also provides a point cloud target detection device, the device includes: a voxel feature extraction module for processing the three-dimensional voxels of the to-be-determined scene through a three-dimensional sparse convolutional neural network grid to obtain the global voxel feature of the scene to be determined, and generate a target proposal frame; point cloud feature extraction module, used to process the original point cloud of the scene to be detected through the point cloud network, and obtain the local part of the scene to be detected point cloud features; a feature fusion module, used to fuse the local point cloud features and the global voxel features based on a preset dual encoder and fusion decoder algorithm, adjust the target proposal frame, and generate a three-dimensional target frame , and completes the detection of point cloud targets in the scene to be detected based on the three-dimensional target frame.

In addition, in order to achieve the above object, the present invention also provides a device for detecting a point cloud target, the device for detecting a point cloud target includes a processor, a memory, and a device that is stored in the memory and can be executed by the processor. A detection program for a point cloud target, wherein when the detection program for a point cloud target is executed by the processor, the steps of the above-mentioned method for detecting a point cloud target are implemented.

In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium on which a detection program of a point cloud target is stored, wherein when the detection program of the point cloud target is executed by a processor, The steps of implementing the method for detecting a point cloud target as described above.

The present invention provides a point cloud target detection method. The method processes the three-dimensional voxel grid of the to-be-determined scene through a three-dimensional sparse convolutional neural network, obtains the global voxel feature of the to-be-determined scene, and generates a target Proposal frame; process the original point cloud of the scene to be detected through the point cloud network, and obtain the local point cloud feature of the scene to be detected; based on the preset dual encoder and fusion decoder algorithm, fuse the local point cloud feature and the The global voxel feature is used to adjust the target proposal frame to generate a three-dimensional target frame, and based on the three-dimensional target frame, the detection of the point cloud target in the scene to be detected is completed. Through the above method, the present invention obtains local point cloud features and global voxel features through point cloud network and three-dimensional sparse convolution network respectively, and generates a fusion multi-modal feature through preset dual encoder and fusion decoder algorithms. The three-dimensional target frame is used to complete the detection of the point cloud target in the scene to be detected. Therefore, using multimodal fusion to determine the three-dimensional target frame improves the accuracy and efficiency of the point cloud target detection method, and solves the technical problems of low accuracy and low efficiency of the current point cloud target detection method.

Description of drawings

1 is a schematic diagram of the hardware structure of a point cloud target detection device involved in an embodiment of the present invention;

2 is a schematic flowchart of a first embodiment of a method for detecting a point cloud target according to the present invention;

FIG. 3 is a schematic diagram of functional modules of a first embodiment of a device for detecting a point cloud target according to the present invention.

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

The point cloud target detection method involved in the embodiments of the present invention is mainly applied to a point cloud target detection device, and the point cloud target detection and generation device may be a PC, a portable computer, a mobile terminal and other devices with display and processing functions.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of a hardware structure of a point cloud target detection device involved in an embodiment of the present invention. In this embodiment of the present invention, a device for detecting a point cloud target may include a processor 1001 (eg, a CPU), a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 . Wherein, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface); the memory 1005 may be a high-speed RAM memory, or a non-volatile memory, such as a disk memory, and the memory 1005 may optionally be a storage device independent of the aforementioned processor 1001 .

Those skilled in the art can understand that the hardware structure shown in FIG. 1 does not constitute a limitation on the detection device of the point cloud target, and may include more or less components than the one shown, or combine some components, or different Component placement.

Continuing to refer to FIG. 1 , the memory 1005 as a computer-readable storage medium in FIG. 1 may include an operating system, a network communication module, and a detection program for point cloud objects.

In FIG. 1, the network communication module is mainly used to connect to the server, and perform data communication with the server; and the processor 1001 can call the detection program of the point cloud object stored in the memory 1005, and execute the point cloud object detection program provided by the embodiment of the present invention. Detection method.

An embodiment of the present invention provides a method for detecting a point cloud target.

Referring to FIG. 2 , FIG. 2 is a schematic flowchart of a first embodiment of a method for detecting a point cloud target according to the present invention.

In this embodiment, the method for detecting the point cloud target includes the following steps:

Step S10, processing the three-dimensional voxel grid of the to-be-determined scene through a three-dimensional sparse convolutional neural network, obtaining the global voxel feature of the to-be-determined scene, and generating a target proposal frame;

In this embodiment, the three-dimensional voxel feature of the empty voxel of the target to be determined is extracted by the three-dimensional sparse convolutional neural network; the three-dimensional voxel feature is compressed into a two-dimensional space by the second encoder to obtain two The dimensional voxel feature is used as the global voxel feature, wherein the second encoder is a three-dimensional sparse convolutional network multilayer encoder.

Further, based on the two-dimensional proposal network, a rough proposal frame of the to-be-determined target is generated according to the classification and regression of the to-be-determined target.

Step S20, processing the original point cloud of the scene to be detected through the point cloud network to obtain local point cloud features of the scene to be detected;

In this embodiment, based on a preset sampling rule, at least one point is selected for sampling in the neighborhood of the target to be determined; the features of the sampling points in the neighborhood are extracted by a multilayer perceptron; The sampling point feature is sampled layer by layer, and the sampling result is used as the local point cloud feature of the target to be determined, wherein the first encoder is a point cloud network multi-layer encoder.

Step S30, based on the preset dual encoder and fusion decoder algorithm, fuse the local point cloud feature and the global voxel feature, adjust the target proposal frame, generate a three-dimensional target frame, and based on the three-dimensional target frame to complete the detection of the point cloud target in the scene to be detected;

In this embodiment, based on the original point cloud, the local point cloud features and the global voxel features are input into a multimodal feature fusion network based on an attention mechanism, and the local point cloud features and the The global voxel feature is used to generate the global feature of the target to be determined; the global feature is fused with the rough proposal frame through a preset fine-tuning network, and the target is determined according to the classification and regression of the target to be determined proposal box.

In a specific embodiment, through the multimodal feature fusion network, fewer voxel features are gradually transferred to more voxel features, and finally the global voxel features are mapped to the original point cloud, The global feature is generated by fusing the local point cloud feature and the global voxel feature.

Region proposal, also called region proposal, is a typical fully convolutional network, so in the process of network training, the back-propagation algorithm and gradient descent algorithm are also applicable to region proposal network

This embodiment provides a point cloud target detection method. The method processes the three-dimensional voxel grid of the to-be-determined scene through a three-dimensional sparse convolutional neural network, obtains the global voxel feature of the to-be-determined scene, and generates The target proposal frame; the original point cloud of the scene to be detected is processed through the point cloud network, and the local point cloud features of the scene to be detected are obtained; based on the preset dual encoder and fusion decoder algorithm, the local point cloud features and all According to the global voxel feature, the target proposal frame is adjusted to generate a three-dimensional target frame, and based on the three-dimensional target frame, the detection of the point cloud target in the scene to be detected is completed. Through the above method, the present invention obtains local point cloud features and global voxel features through point cloud network and three-dimensional sparse convolution network respectively, and generates a fusion multi-modal feature through preset dual encoder and fusion decoder algorithms. The three-dimensional target frame is used to complete the detection of the point cloud target in the scene to be detected. Therefore, using multimodal fusion to determine the three-dimensional target frame improves the accuracy and efficiency of the point cloud target detection method, and solves the technical problems of low accuracy and low efficiency of the current point cloud target detection method.

Based on the embodiment shown in FIG. 2 above, in this embodiment, the step S10 includes:

Divide the point cloud space to be detected into a regular voxel expression form, in which only the voxel grid containing points is retained, and the rest of the voxels are considered as empty voxels;

For each empty voxel, the average feature of the points it contains is used as its initial input feature, and then the three-dimensional sparse convolutional neural network is used to perceive the voxel grid and give a rough proposal frame of the image to be determined. .

Voxel is the abbreviation of Volume Pixel, and a volume containing voxels can be represented by volume rendering or polygon isosurface extraction of a given threshold outline. As its name suggests, it is the smallest unit of digital data in 3D space segmentation, and voxels are used in 3D imaging, scientific data and medical imaging. Conceptually similar to the smallest unit of two-dimensional space - pixel, pixel is used in the image data of two-dimensional computer image. Some true 3D displays use voxels to describe their resolution, for example a display that can display 512 x 512 x 512 voxels.

In this embodiment, a three-dimensional sparse convolutional network multi-layer encoder is constructed to extract global position information and propose a target frame. First, the point cloud space is divided into a regular voxel representation, in which only the voxel grid containing the points is kept, and the rest of the voxels are considered as empty voxels. Specifically, for each of the empty voxels, the average feature of the points it contains is used as its initial input feature. Sparse 3D convolution is then employed to efficiently perceive the voxel space and give object proposals. It includes the following two sub-steps:

For 3D sparse convolution, in order to reduce memory usage and improve running speed, 3D sparse convolution is used to replace the original 3D convolution, that is, only empty voxels are used for feature extraction and empty voxels are ignored. The Spconv library is used to stack 4 layers of sparse convolution modules, where each sparse convolution module contains two layers of sub-stream convolution and one layer of sparse convolution module for stepwise downsampling. Assuming that the input voxel tensor is represented as L×W×H×C, then the network output of the three-dimensional sparse convolution can be expressed as

随后将其输入用于鸟瞰图目标检测的二维RPN网络，该网络由四层二维卷积神经网络搭建，采用U-Net网络结构，逐层的输出具体表示为

各层均采用3×3卷积以减小学习参数，得到进一步细化的特征图用于后续的预测。最后通过基于锚框的分类和回归得到RPN网络检测出的物体提议，即对每一个像素点都预先设置一个对应的锚框，对该锚框做微调以实现精准的定位，因此对于大小为

的特征图，一共会有

个预定义框。值得注意的是在自动驾驶场景中，物体的三维框可以表示为{x,y,z,l,w,h,r}，各个参数依次表示为物体框中心位置、物体的大小和针对x-y平面的旋转角，而不需要考虑其他的角度自由度，大大降低了预测的难度。2D Region Proposal Network (RPN) Classification and Object Proposal Regression. The three-dimensional output features are compressed from high-dimensional to two-dimensional space to obtain a two-dimensional image representation as

Then it is input into a two-dimensional RPN network for bird's-eye view target detection. The network is built by a four-layer two-dimensional convolutional neural network and adopts a U-Net network structure. The output of each layer is specifically expressed as

Each layer adopts 3×3 convolution to reduce the learning parameters, and obtain further refined feature maps for subsequent prediction. Finally, the object proposal detected by the RPN network is obtained through the classification and regression based on the anchor frame, that is, a corresponding anchor frame is preset for each pixel point, and the anchor frame is fine-tuned to achieve accurate positioning. Therefore, for the size of

The feature map of , there will be a total of

predefined boxes. It is worth noting that in the automatic driving scene, the three-dimensional frame of the object can be expressed as {x, y, z, l, w, h, r}, and each parameter is expressed as the center position of the object frame, the size of the object, and the xy plane. The rotation angle does not need to consider other angular degrees of freedom, which greatly reduces the difficulty of prediction.

Based on the embodiment shown in FIG. 2 above, in this embodiment, the step S20 includes:

In this embodiment, a point cloud network and a three-dimensional convolutional neural network are used to process the original point cloud and the three-dimensional voxel grid respectively, so as to extract local point cloud features of the scene and global rough object spatial position information. First, given the original point cloud of the scene, uniformly collect a fixed number N as the input of the initial point cloud branch, and voxelize it to generate a sparse voxel grid as the input of the convolution branch, and then pass the local point cloud features The extractor and the 3D sparse convolutional neural network separately process the scene to extract local point cloud features and global rough localization information, which are then input into the multimodal feature fusion network, which adaptively aggregates the advantages of both to facilitate subsequent object detection task. Includes the following sub-steps:

每一层点集特征提取模块的输入依次为上一层的输出，而当前层的输出为后续FPS采样的点集。通过点集特征提取模块提取各层点云的局部点云特征，且逐层扩大以得到全局点云语义信息。设点云中存在点p_i，设S_i为在以点p_i为中心，半径为r的球邻域内部的点所构成的集合。p_i的输出特征的计算方法如下：A point cloud network multi-layer encoder is constructed to extract spatial geometric structure information, and a fixed point cloud number N is uniformly sampled from the original point cloud data by using the farthest point sampling strategy.

The input of the point set feature extraction module of each layer is the output of the previous layer in turn, and the output of the current layer is the point set of the subsequent FPS sampling. The local point cloud features of each layer of point cloud are extracted through the point set feature extraction module, and the global point cloud semantic information is obtained by expanding layer by layer. Suppose that there is a point p _i in the point cloud, and let S _i be a set of points within a spherical neighborhood with the point p _i as the center and the radius r. The output features of _pi are calculated as follows:

Randomly sample _k points from the set Si to form the set _Si ;

The feature fusion extraction is performed on the points sampled in the neighborhood through the multi-layer _perceptron , and the output feature of the point pi is calculated. Calculated as follows:

First, the multi-layer perceptron is used to further extract the features of k points in the point local area to obtain the corresponding _n -dimensional map features, and then use the maximum pooling process in the feature dimension to obtain the neighbors with pi as the center and r as the radius The maximum information representation of the domain, and then the multi-layer perceptron is used to further abstract the _n -dimensional high-dimensional features of the maximum information representation to obtain the output features of point pi.

The point cloud is repeatedly sampled by FPS layer by layer to the corresponding number of points, and the local point cloud feature extraction of the original point cloud of the scene to be detected is realized by aggregating the neighborhood features of the sampled points.

FPS sampling: Farthest Point Sampling is a very common sampling algorithm. It is widely used because it can ensure uniform sampling of samples. For example, PointNet++ in the 3D point cloud deep learning framework performs FPS sampling on sample points. Reclustering is used as the receptive field. The 3D target detection network VoteNet performs FPS sampling on the scattered points obtained by voting and then performs clustering. The 6D pose estimation algorithm PVN3D is used to select 8 feature points of the object to vote and calculate the pose.

In a specific embodiment, Multilayer Perception (MLP, Multilayer Perception) is a feedforward artificial neural network model, which maps multiple input data sets to a single output data set.

This embodiment provides a point cloud target detection method. The method processes the three-dimensional voxel grid of the to-be-determined scene through a three-dimensional sparse convolutional neural network, obtains the global voxel feature of the to-be-determined scene, and generates The target proposal frame; the original point cloud of the scene to be detected is processed through the point cloud network, and the local point cloud features of the scene to be detected are obtained; based on the preset dual encoder and fusion decoder algorithm, the local point cloud features and all According to the global voxel feature, the target proposal frame is adjusted to generate a three-dimensional target frame, and based on the three-dimensional target frame, the detection of the point cloud target in the scene to be detected is completed. Through the above method, the present invention obtains local point cloud features and global voxel features through point cloud network and three-dimensional sparse convolution network respectively, and generates a fusion multi-modal feature through preset dual encoder and fusion decoder algorithms. The three-dimensional target frame is used to complete the detection of the point cloud target in the scene to be detected. Therefore, using multimodal fusion to determine the three-dimensional target frame improves the accuracy and efficiency of the point cloud target detection method, and solves the technical problems of low accuracy and low efficiency of the current point cloud target detection method.

Based on the embodiment shown in FIG. 2 above, in this embodiment, the step S30 includes:

Adaptive fusion of multimodal-aware features and fine-tuning of object proposals. For the local point cloud information of the scene extracted in step 1 and the global rough object spatial position information, in order to effectively exert their respective advantages, the attention mechanism method is used to effectively fuse the two features based on the original point cloud. Then, combined with the preliminarily predicted target proposal frame, the target frame is fine-tuned by fused point cloud features of local objects. Specifically include the following steps:

Adaptive fusion of multimodal perception features based on attention mechanism. According to the multi-scale point cloud and voxel features extracted in step 1, the present invention designs a dual encoder-decoder structure, and fuses the two based on the point cloud. Includes the following sub-steps:

和

通过三最近邻插值函数赋值，设S(v_j)表示在欧式空间中距离V_j最近的三个点组成的组合，P_k表示其中一点。Mapping of voxel features to point cloud space. For point cloud features and voxel features of the same scale, considering that the point cloud retains better spatial structure and position information, the present invention is based on the point cloud and uses the nearest neighbor interpolation method to map the voxel features to the point cloud. superior. Let the outputs of the point cloud branch and the voxel branch in step 1 be

and

Through the assignment of the three-nearest-neighbor interpolation function, let S(v _j ) represent the combination of three points closest to V _j in the Euclidean space, and P _k represents one of the points.

Here w _kj is calculated as follows:

Each point-based voxel feature is obtained by the weighted summation of Euclidean distances of the three voxel features in the surrounding field.

Specifically, the nearest neighbor interpolation method (Nearest_Neighbor) assigns the gray value of the most adjacent pixel of the original pixel in the transformed image to the original pixel, which is the simplest gray value interpolation. Also known as zero-order interpolation, it is to make the gray value of the transformed pixel equal to the gray value of the nearest input pixel.

Feature fusion based on attention mechanism. For different positions in the scene, different features play different roles, such as the sidewall of the car, the column in the scene and other positions that do not contain detailed features, the voxel expression is convenient and fast to locate the object, such as the tire of the car , human posture and other positions containing discriminative features, using point cloud features to better describe objects and help in the adjustment stage for fine regression and category judgment. In order to finely fuse different features, the structure of the decoder is gradually fused from coarse to fine, and the features of the minority point cloud set are transferred to the majority point cloud set by means of interpolation with the three nearest neighbors, and finally restored to the original point. Cloud, obtains multi-modal strong representational fusion features about the entire scene, and feeds them into the subsequent fine-tuning network to further improve the detection accuracy.

Given the fusion feature and target proposal frame, a fine-tuning network is designed to further accurately regress the spatial information of the object, and filter out possible falsely detected objects. For each target proposal frame, the point set G _i in it is taken, and the global feature is extracted by maximum pooling, and the output feature of the object G _i is calculated. Calculated as follows:

Then the precise information of the object is obtained through the confidence prediction and the target box regression branch, respectively.

Confidence prediction is defined as the probability of a predicted value within the allowable error range obtained by using interval estimation in mathematical statistics when estimating future conditions.

Loss function design. The loss function is used to supervise the deep neural network to learn the features required for the point cloud target detection task. The supervision here includes classification supervision and regression supervision; the classification loss function adopts the following cross entropy loss function:

where P(a _i ) represents the predicted score of the i-th preset box, and Q(a _i ) represents the real label value of the preset box. The regression loss function of the present invention adopts the Smooth-L1 loss function:

where υ represents a variable of the three-dimensional box {x, y, z, l, w, h, r}. Through the joint supervision of the classification loss function and the regression loss function, the network can finally learn the ability of point cloud object detection.

This embodiment provides a point cloud target detection method. The method processes the three-dimensional voxel grid of the to-be-determined scene through a three-dimensional sparse convolutional neural network, obtains the global voxel feature of the to-be-determined scene, and generates The target proposal frame; the original point cloud of the scene to be detected is processed through the point cloud network, and the local point cloud features of the scene to be detected are obtained; based on the preset dual encoder and fusion decoder algorithm, the local point cloud features and all According to the global voxel feature, the target proposal frame is adjusted to generate a three-dimensional target frame, and based on the three-dimensional target frame, the detection of the point cloud target in the scene to be detected is completed. Through the above method, the present invention obtains local point cloud features and global voxel features through point cloud network and three-dimensional sparse convolution network respectively, and generates a fusion multi-modal feature through preset dual encoder and fusion decoder algorithms. The three-dimensional target frame is used to complete the detection of the point cloud target in the scene to be detected. Therefore, using multimodal fusion to determine the three-dimensional target frame improves the accuracy and efficiency of the point cloud target detection method, and solves the technical problems of low accuracy and low efficiency of the current point cloud target detection method.

In addition, an embodiment of the present invention also provides a device for detecting a point cloud target.

Referring to FIG. 3 , FIG. 3 is a schematic diagram of functional modules of the first embodiment of the device for detecting a point cloud target according to the present invention.

In this embodiment, the detection device for the point cloud target includes:

The voxel feature extraction module 10 is configured to process the three-dimensional voxel grid of the to-be-determined scene through a three-dimensional sparse convolutional neural network, obtain the global voxel feature of the to-be-determined scene, and generate a target proposal frame;

The point cloud feature extraction module 20 is used to process the original point cloud of the scene to be detected through the point cloud network, and obtain the local point cloud features of the scene to be detected;

The feature fusion module 30 is configured to fuse the local point cloud feature and the global voxel feature based on a preset dual encoder and fusion decoder algorithm, adjust the target proposal frame, generate a three-dimensional target frame, and generate a three-dimensional target frame based on The three-dimensional target frame completes the detection of point cloud targets in the scene to be detected.

Further, the voxel feature extraction module 10 specifically includes:

a voxel division unit, which is used to divide the point cloud space to be detected into a regular voxel expression form, in which only the voxel grid containing points is retained, and the rest of the voxels are regarded as empty voxels;

A rough proposal box generation unit is used for each of the empty voxels to take the average feature of the points it contains as its initial input feature, and then use the three-dimensional sparse convolutional neural network to perceive the voxel grid and give the A rough proposal box describing the image to be determined.

Further, the point cloud feature extraction module 20 specifically includes:

每一层局部点云特征提取模块的输入依次为上一层的输出，而当前层的输出为后续最远点采样的点集；The sampling unit is used to uniformly sample the fixed point cloud from the original point cloud by using the farthest point sampling strategy to be N,

The input of the local point cloud feature extraction module of each layer is the output of the previous layer in turn, and the output of the current layer is the point set of the subsequent furthest point sampling;

The local point cloud feature extraction unit is used to extract the local point cloud features of each layer of the point cloud of the scene to be detected through the local point cloud feature extraction module, and expand layer by layer to obtain the global point cloud semantic information. For a point _pi , let Si be the set of points in the neighborhood of the sphere with the point _{pi as} the center and the radius r.

Further, the detection device for the point cloud target also includes:

The fine-tuning network adjustment unit is used to give the fusion feature and target proposal frame, design the fine-tuning network to further accurately return the spatial information of the object, and filter out the possible falsely detected objects.

Wherein, each module in the above-mentioned device for detecting a point cloud target corresponds to each step in the above-mentioned embodiment of the above-mentioned method for detecting a point cloud target, and the functions and implementation processes thereof will not be repeated here.

In addition, an embodiment of the present invention further provides a computer-readable storage medium.

The computer-readable storage medium of the present invention stores a point cloud target detection program, wherein when the point cloud target detection program is executed by the processor, the steps of the above point cloud target detection method are implemented.

The method implemented when the detection program of the point cloud target is executed may refer to the various embodiments of the detection method of the point cloud target of the present invention, which will not be repeated here.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or system comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or system. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system that includes the element.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

The present application may be used in numerous general purpose or special purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, including A distributed computing environment for any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on such understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM) as described above. , magnetic disk, optical disk), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields , are similarly included in the scope of patent protection of the present invention.

Claims (10)

1. A method for detecting a point cloud target is characterized by comprising the following steps:

processing the three-dimensional voxel grid of the scene to be determined through a three-dimensional sparse convolution neural network to obtain the global voxel characteristics of the scene to be determined and generate a target proposal frame;

processing an original point cloud of a scene to be detected through a point cloud network to obtain local point cloud characteristics of the scene to be detected;

and fusing the local point cloud characteristics and the global voxel characteristics based on a preset double-encoder and fusion decoder algorithm, adjusting the target proposal frame to generate a three-dimensional target frame, and completing the detection of the point cloud target in the scene to be detected based on the three-dimensional target frame.

2. The method for detecting the point cloud target of claim 1, wherein the processing the three-dimensional voxel grid of the scene to be determined through a three-dimensional sparse convolutional neural network to obtain a global voxel characteristic of the scene to be determined and generate a target proposal box comprises:

dividing the point cloud space to be detected into regular voxel expression forms, wherein only a voxel grid containing points is reserved, and the rest voxels are considered as empty voxels;

and for each empty voxel, adopting the average characteristic of the contained points as the initial input characteristic thereof, then adopting the three-dimensional sparse convolution neural network to sense the voxel grid and giving a target proposal frame of the image to be determined.

3. The method for detecting point cloud target according to claim 2, wherein said step of adopting, for each empty voxel, the average feature of the points contained in the empty voxel as its initial input feature, and then adopting the three-dimensional sparse convolutional neural network to sense the voxel grid and give a rough proposed box of the image to be determined comprises:

the three-dimensional output characteristics are compressed to a two-dimensional space in a height dimension manner to obtain a two-dimensional image which is characterized in that

Inputting the two-dimensional image representation into a two-dimensional RPN network for aerial view target detection, and specifically expressing the layer-by-layer output as

Each layer adopts 3 multiplied by 3 convolution to reduce learning parameters, and further refined characteristic graphs are obtained for subsequent prediction, wherein the two-dimensional RPN network is built by four layers of two-dimensional convolution neural networks and adopts a U-Net network structure;

the object proposal detected by the RPN is obtained through classification and regression based on the anchor frame, namely, a corresponding anchor frame is preset for each pixel point, and the anchor frame is finely adjusted to realize accurate positioning, so that the object proposal with the size of being equal to that of the object proposal is obtained

All together have

A predefined frame.

4. The method for detecting a point cloud target according to claim 2, wherein the dividing the point cloud space to be detected into regular voxel expression forms, wherein only a voxel grid containing points is retained, and the rest voxels are regarded as empty voxels, comprises:

stacking 4 layers of the three-dimensional sparse convolution modules by using a Spconv library, wherein each sparse convolution module comprises two layers of sub-flow convolution and one layer of sparse convolution module for gradually down-sampling; assuming that the input voxel tensor is expressed as L × W × H × C, the network output of the three-dimensional sparse convolution can be expressed as

5. The method for detecting a point cloud target according to claim 1, wherein the processing an original point cloud of a scene to be detected through a point cloud network to obtain a local point cloud characteristic of the scene to be detected comprises:

uniformly sampling a fixed point cloud number from the original point cloud by utilizing a furthest point sampling strategy

The input of each layer of local point cloud feature extraction module is the output of the previous layer in sequence, and the output of the current layer is a point set sampled by the subsequent farthest point;

extracting local point cloud characteristics of each layer of point cloud of a scene to be detected through the local point cloud characteristic extraction module, expanding layer by layer to obtain global point cloud semantic information, and setting a point p in the point cloud _i Is provided with S _i At a point p _i The set of points inside a sphere neighborhood of radius r, centered.

6. The method for detecting point cloud target of claim 5, wherein the local point cloud feature extraction module extracts local point cloud features of each layer of point cloud of the scene to be detected, and expands layer by layer to obtain global point cloud semantic information, and the point p is set to exist in the point cloud _i Is provided with S _i At a point p _i A set of centered, r-radius points within a spherical neighborhood comprising:

from the set S _i K points are randomly sampled to form set S' _i ；

Performing feature fusion extraction on the sampled points in the neighborhood through a multilayer perceptron, and calculating to obtain a point p _i The calculation formula is as follows:

f(p _i )＝MLP(max _{j＝1，...，k} {MLP(f _j )}) (1)

firstly, a multilayer perceptron is adopted to further extract the features of k points in a point local area to obtain the corresponding n-dimensional mapping feature, and then the maximum pooling processing is adopted in the feature dimension to obtain the p-dimensional mapping feature _i The maximum information representation of the neighborhood with the radius as the center and r is adopted, and then the n-dimensional high-dimensional feature of the maximum information representation is further abstracted by adopting a multilayer perceptron to obtain a point p _i The output characteristics of (1);

and repeatedly sampling the point cloud to corresponding points layer by layer, aggregating neighborhood characteristics of the sampled points, and extracting the high-dimensional geometric characteristics of the point cloud information.

7. The method for detecting a point cloud target according to claim 1, wherein the fusing the local point cloud feature and the global voxel feature based on a preset dual encoder and fusion decoder algorithm, adjusting the target proposal frame, generating a three-dimensional target frame, and after completing the detection of the point cloud target in the scene to be detected based on the three-dimensional target frame, the method further comprises:

giving a fusion characteristic and a target proposing box, designing a fine tuning network to further accurately regress the spatial information of the object, and filtering out the possible false detection object;

for each of the goal proposals, a set of points G is framed therein _i And extracting global features by adopting maximum pooling, and calculating to obtain an object G _i The calculation formula is as follows:

f(G _i )＝MLP(max _{j＝1，...，k} {MLP(f _j )}) (2)

and respectively acquiring the accurate information of the scene to be detected through confidence prediction and regression branches of the target proposal box.

8. A point cloud target detection device is characterized by comprising:

the voxel characteristic extraction module is used for processing the three-dimensional voxel grid of the scene to be determined through a three-dimensional sparse convolution neural network to obtain the global voxel characteristic of the scene to be determined and generate a target proposal frame;

the point cloud feature extraction module is used for processing the original point cloud of the scene to be detected through a point cloud network to obtain the local point cloud feature of the scene to be detected;

and the feature fusion module is used for fusing the local point cloud features and the global voxel features based on a preset double-encoder and fusion decoder algorithm, adjusting the target proposal frame to generate a three-dimensional target frame, and completing the detection of the point cloud target in the scene to be detected based on the three-dimensional target frame.

9. A detection apparatus of a point cloud object, characterized in that the detection apparatus of the point cloud object comprises a processor, a memory, and a detection program of the point cloud object stored on the memory and executable by the processor, wherein the detection program of the point cloud object, when executed by the processor, implements the steps of the detection method of the point cloud object according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a detection program of a point cloud object, wherein the detection program of the point cloud object, when executed by a processor, implements the steps of the detection method of the point cloud object according to any one of claims 1 to 7.

Images