CN102737236B - Method for automatically acquiring vehicle training sample based on multi-modal sensor data - Google Patents

Abstract

A method for automatically obtaining vehicle training samples based on multi-modal sensor data, the vehicle detection step based on laser and positioning data: according to the distance, angle and laser sensor calibration parameters of the laser data, the two-dimensional coordinates relative to the data acquisition vehicle are obtained, and the Describe the contour information of the object level; through the analysis of the shape, and the detection and tracking of the moving object, extract the time series of parameters such as the position and direction of the candidate vehicle relative to the data collection vehicle; the visual image sample extraction step: according to the candidate vehicle at each moment According to the geometric relationship between the laser sensor and the image acquisition device, the candidate vehicle is projected into the image to generate a region of interest, and the detector is used to correct the region of interest. For each candidate vehicle, according to its position and direction, etc. The parameter calculates the relative angle of view of the candidate vehicle relative to the camera, removes image frame samples with similar angles of view, and automatically extracts sample pictures of the candidate vehicle under different angles of view.

Description

A method for automatically obtaining vehicle training samples based on multimodal sensor data

technical field

The invention relates to the technical fields of computer vision, robot and machine learning, in particular to a method for automatically acquiring vehicle training samples based on multimodal sensor data.

Background technique

Vehicle detection is an important problem in the field of automotive driver assistance systems (ADAS). There has been a large amount of related research in the field of vehicle detection, which has proved that vehicles can be detected using lasers, radars, monocular/stereo cameras, and multi-sensor fusion.

Due to the low cost of monocular cameras and the simple calibration problem, detection methods based on monocular vision have been extensively studied in the fields of computer vision and robotics. When using visual sensors, the appearance of the vehicle itself and the appearance of the vehicle at different angles are very different, which brings great difficulties to detection. Recently, more and more researchers try to use machine learning methods to detect vehicles.

In these methods, the detector is pre-trained using a series of sample images. There are many datasets open for training detectors. PASCAL provides many standardized datasets for object detection.

Among them, the UIUC data set is a data set specially used for vehicle detection and recognition, including 550 vehicle pictures with a resolution of 100×40 as training positive samples, and contains two test sets: 170 uniform images with the same resolution as the training positive samples. scale vehicles, and 108 images containing 139 multi-scale vehicles.

This dataset is used in many studies to illustrate research findings. However, all the vehicles in the UIUC data set are images from the side view. In road vehicle detection, most of the detected vehicles are from the front or rear view, so this data set is not applicable.

Another disadvantage is that the pictures in the UIUC dataset are all black and white pictures, and the use of this dataset has a great limitation on the feature space of the detector. Unlike the UIUC dataset, the MIT dataset includes 516 positive sample images with a resolution of 128×128, all of which are from the front or rear perspective.

For methods that perform well today, training samples are an important factor affecting their performance. In order to study the establishment of a multi-view vehicle detector, USC researchers established a dataset of multi-view vehicle sample images and test images. The dataset includes 1028 positive images of vehicles from various angles with a resolution of 128×64, and 196 test images containing 410 vehicles of different scales and angles.

However, all the samples in this dataset do not contain the pose information of the vehicle. Usually, the training data needs to be manually labeled and classified according to the training requirements, which greatly limits the number and performance of samples. This becomes a bottleneck restricting the development of algorithms. The performance of the detector is usually unstable to changes in the environment.

Contents of the invention

The purpose of the present invention is to provide a method for automatically acquiring vehicle training samples based on multi-modal sensor data, which relates to a method for automatically generating multi-angle vehicle sample images and including pose information.

The invention discloses a method for automatically acquiring vehicle training samples based on multi-modal sensor data, comprising the following steps:

Vehicle detection steps based on laser and positioning data: According to the distance, angle and laser sensor calibration parameters of the laser data, obtain the two-dimensional coordinates of the vehicle relative to the data acquisition to describe the contour information of the object level; through the analysis of the shape, and the moving object The detection and tracking of the candidate vehicle, extracting the time series of parameters such as the position and direction of the candidate vehicle relative to the data collection vehicle;

Visual image sample extraction step: According to the position and direction of the candidate vehicle at each moment, according to the geometric relationship between the laser sensor and the image acquisition device, project the candidate vehicle into the image, generate the region of interest, and use the detector to correct In the area of interest, for each candidate vehicle, calculate the relative angle of view of the candidate vehicle relative to the camera according to its position and direction parameters, remove image frame samples with similar angles of view, and automatically extract sample pictures of the candidate vehicle under different angles of view.

Further, as a preference, the vehicle detection step based on laser and positioning data further includes:

Data Fusion: Fusion of data from the same or near time from each laser sensor;

Clustering: cluster the data from each laser sensor according to the distance between two adjacent points;

Labeling: distinguishing clusters into three types: stationary objects, moving objects, or uncertain; map generation: generating a map describing the static environment around the moving track of the data collection vehicle;

Detection: find candidate vehicles in the current classified laser fusion data;

Tracking: Associating detection results with previous tracking results, updating tracking status and car body and motion parameters;

Validation: Validate objects by tracking their motion and shape information.

Further, as a preference, the visual image sample extraction step further includes:

Region of interest extraction based on laser data: According to the position and direction of the candidate vehicle at each moment, according to the geometric relationship between the laser sensor and the image acquisition device, the candidate vehicle is projected into the image, and the region of interest containing the candidate vehicle is extracted ;

Correction of the region of interest based on image technology: use the image-based detection method to correct the region of interest and find the candidate vehicle;

Vehicle sample image extraction and deduplication: According to the correction results, image frame samples with similar viewing angles are removed, and sample pictures of candidate vehicles under different viewing angles are automatically extracted.

Further, as a preference, the multi-modal sensor includes: a multi-view laser sensor, a multi-view image acquisition device and a positioning system; the multi-view laser sensor and the multi-view image acquisition device are used to monitor the surrounding environment of the data acquisition vehicle, Constitute coverage on multiple viewing angles.

Further, as a preference, the positioning system is used to measure the pose of the vehicle with 6 degrees of freedom.

Further, as a preference, the detector used for correcting the region of interest adopts an image-based detection method.

Further, as a preference, the tracking is specifically as follows: for those detection results that cannot be associated with the tracking results, they are regarded as new tracking vehicles; and those tracking vehicles that are not associated with the detection results are considered to have disappeared in the vehicle's Monitoring scope, removed from trace results.

Further, as a preference, the verification specifically includes: if a tracked object does not move within a period of time, it is regarded as stationary and integrated into the map information; if the tracked object’s movement and shape information occur within a short period of time If there are irregular changes, the results are removed; only those tracking results with normal changes in motion and shape are considered as candidate vehicles.

Further, as a preference, the correction of the region of interest specifically includes: according to the pose of the candidate vehicle in the region of interest, selecting a detector based on vehicle training with a specific pose to correct it.

Further, as a preference, the removing duplicate samples specifically includes: screening sample images that appear to be the same or similar according to the moving direction and head orientation of the tracked vehicle relative to the data collection vehicle.

The invention automatically generates multi-angle vehicle sample images and includes pose information through multi-modal sensor data, can effectively avoid manual operations, and provides more freedom and less restrictions for the research of detection algorithms. Moreover, the automatic generation of training samples makes online training possible, making it possible to automatically improve the classifier to adapt to changes in lighting and other environments.

The invention is fully automatic without manual intervention, can obtain a large number of vehicle samples, enriches the training set, and obtains training pictures containing pose information of vehicles, which is convenient for training classifiers based on different poses.

Description of drawings

A more complete and better understanding of the invention, and many of its attendant advantages, will readily be learned by reference to the following detailed description when considered in conjunction with the accompanying drawings, but the accompanying drawings illustrated herein are intended to provide a further understanding of the invention and constitute A part of the present invention, the exemplary embodiment of the present invention and its description are used to explain the present invention, and do not constitute an improper limitation of the present invention, wherein:

Fig. 1 is a flowchart of an embodiment of the present invention;

Fig. 2 is a flow chart of an embodiment of the vehicle detection step based on laser light and positioning data;

Fig. 3 is a flowchart of an embodiment of visual image sample extraction.

Detailed ways

Embodiments of the present invention will be described with reference to FIGS. 1 to 3 .

In order to make the above objects, features and advantages more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 1, a method for automatically obtaining vehicle training samples based on multimodal sensor data includes the following steps:

S1. The vehicle detection step based on laser and positioning data: according to the distance and angle of the laser data and the calibration parameters of the laser sensor, obtain the two-dimensional coordinates of the vehicle relative to the data collection to describe the contour information of the object level; through the analysis of the shape, and Detection and tracking of moving objects to extract candidate vehicles;

S2. Visual image sample extraction step: according to the position and direction of the candidate vehicle at each moment, according to the geometric relationship between the laser sensor and the image acquisition device, project the candidate vehicle into the image to generate an area of interest, and use the detector To modify the region of interest, for each candidate vehicle, calculate the relative viewing angle of the candidate vehicle relative to the camera according to its position, direction and other parameters, remove the image frame samples with similar viewing angles, and automatically extract the samples of the candidate vehicle under different viewing angles picture.

The present invention is to establish a system for automatically generating a vehicle sample data set for training a visual vehicle detector. The data set includes multi-angle vehicle sample images, and each image includes its pose information, so that it is possible to train vehicle detectors for vehicles from different angles.

Multimodal sensors include laser sensors, image acquisition devices, and positioning systems. Laser sensors include laser scanners, laser rangefinders, etc., and image acquisition devices can use cameras or camera systems integrating single or multiple cameras. The purpose of the positioning system is to obtain the position information of the object loaded with the system equipment, such as GPS global satellite positioning system, Galileo satellite positioning system, Beidou satellite positioning system, etc.

Each sensor together forms a multi-view coverage around the data collection vehicle. According to the actual needs and the limitations of the data collection platform, different sensor installations can be selected to achieve detection covering different viewing angles.

Sensor system:

The invention discloses a vehicle sensor system. The system includes three sensors: laser scanner, video camera and GPS/IMU. GPS/IMU is an industrial device used to measure the 6-DOF pose (3D position and angle) of a vehicle. Lasers and cameras are used to monitor the environment around the vehicle, and each sensor can form a full range of coverage. Three Hokuyo UTM-30LX laser scanners are installed in the front left, front right, and rear middle of the vehicle to form a horizontal and all-round coverage. However, the monitoring distance of the Hokuyo UTM-30LX laser scanner is relatively short, usually only in the outdoor traffic environment It can reach 25m, so the SICK LMS291 laser is used in the center of the front to cover and monitor a semicircle area with a radius of 45m. A full range of video sensor coverage can be achieved using multiple cameras. Here, a Ladybug camera integrated with multiple cameras is used, and the collection results are fused with pictures from 6 cameras to form a panoramic image. In order to reduce occlusion, the Ladybug camera is mounted on the top of the vehicle. Sensor data was acquired using two time-synchronized computers: one for laser scanner data and GPS/IMU data acquisition, and the other for Ladybug video data acquisition. For each frame of data, the computer time at the time of recording is given as a timestamp. The delay during data transmission is regarded as a constant value, which can be obtained by testing in advance. After the sensor is calibrated, all the laser scanner data are transformed and transmitted to the same coordinates, here are the local coordinates of the data collection vehicle, and the processing results of the laser data are projected onto the panoramic image to extract image sample pictures.

Processing flow:

As shown in Figure 1, the present invention comprises two steps:

S1. Vehicle detection based on laser and positioning data;

S2. Visual sample image extraction.

Laser scanners are able to directly measure the distance value of an object. According to the angle and sensor calibration parameters, two-dimensional coordinates relative to the vehicle can be obtained to describe the contour information of the object level. Through shape analysis and detection and tracking of moving objects, the time series of parameters such as the position and direction of the candidate vehicle relative to the data collection vehicle can be quickly extracted.

According to the geometric relationship between the laser and the camera, these candidate vehicles are then projected into the panoramic image to generate a region of interest, which also contains the pose information of the vehicle at the current moment.

However, the points scanned by the laser on the object are sparse, and reflection failures will occur on special colored materials. Especially in dynamic traffic environments, where there are many occlusions, the observation of surrounding vehicles may only be partial. This brings great challenges to laser data processing, and there are some errors in the processing results.

Unlike laser data, video image data contains rich information that can be used to refine regions of interest based on laser processing results. In the present invention, a detector based on HOG features is used to refine the region of interest.

Furthermore, during the tracking of a candidate vehicle, its apparent change in the image is slow. When extracting vehicle images from all images, a large number of images with similar perspectives will be generated, which requires a process of selecting vehicle images in different poses.

As shown in Figure 2, the vehicle detection step based on laser light and positioning data further includes:

S11. Data fusion: fusion of data from the same or near time from each laser scanner;

S12. Clustering: clustering the data from each laser sensor according to the distance between two adjacent points;

S13. Labeling: distinguishing clusters into three types: stationary objects, moving objects, or uncertain types;

S14. Map generation: generate a map describing the static environment around the moving track of the data collection vehicle;

S15. Detection: find candidate vehicles in the current classified laser fusion data;

S16. Tracking: Associating the detection results with the previous tracking results, updating the tracking status and vehicle body and motion parameters;

S17. Verification: verify the object by tracking its motion and shape information.

The above working process will be described in detail below.

S1. Vehicle detection based on laser and positioning data:

In the present invention, a method for detecting and tracking road vehicles based on multiple single-line laser scanner data and positioning data is disclosed. According to the distance and angle of the laser data and the calibration parameters of the laser sensor, the two-dimensional coordinates of the vehicle relative to the data acquisition are obtained. Describe the contour information of the object level; through the analysis of the shape and the detection and tracking of the moving object, the time series of parameters such as the position and direction of the candidate vehicle relative to the data collection vehicle are extracted. This data provides parameters such as the position and direction of the candidate vehicle relative to the data collection vehicle at each moment for S2 visual sample image extraction. The vehicle detection framework based on laser scanner data and positioning data is shown in Figure 2, and each module is introduced below.

S11. Data fusion: The data from the same or near time from each laser scanner is fused. In order to reduce the use of memory and save the angle information of the data, the fused data records the distance data from different laser sensors in sequence. Each distance data obtains angle information according to its sequence, and at the same time, according to the calibration parameters between lasers, it can be converted into two-dimensional coordinates relative to the data collection vehicle. The fused laser data can describe the contour information of the object as viewed from the perspective of the data collection vehicle.

S12. Clustering: clustering the data from each laser sensor according to the distance between two adjacent points. The Euclidean distance between two points is used here, taking into account the angular interval. If the distance is greater than a given threshold, a new cluster is generated. A cluster can be an observation of an object, which may be moving or stationary, where the clustering is only performed on the same laser sensor data.

S13. Labeling: distinguishing the clusters into three types: stationary objects, moving objects, or uncertain. First, according to the location information recorded by the GPS/IMU of the data collection vehicle, each cluster is projected into the global coordinate system. For each cluster, if it can match the expected estimate of the static environment of the previous frame, it can be considered stationary, if it matches the expected estimate of a moving object, it is considered moving, otherwise it is considered uncertain of. Moreover, clusters can be supplemented by classification based on prior knowledge, such as vehicle size, road geometry information, etc. The classified laser data will be applied in map generation and moving object detection modules.

S14. Map generation: generate a map describing the static environment around the moving track of the data collection vehicle. The map is described by a grid, and the value of each grid indicates the probability that the grid is occupied by an object.

S15. Detection: find candidate vehicles in the current classified laser fusion data. Here locally observed and coincident observations are the two difficult parts in detection. In order to improve the accuracy of feature parameter estimation and reduce false detection, it is necessary to combine the clustering results. In the present invention, the model of the vehicle is simply defined by the outline box of the vehicle, and the algorithm of cluster merging and model estimation is developed. In addition, the expected estimation of the results of previous detection tracks can help reduce errors in cluster merging, especially for clusters that share the same vehicle but are not consecutive.

S16. Tracking: Associating the detection results with the previous tracking results, updating the tracking status and vehicle body and motion parameters. For those detection results that do not find correlative tracking results, they are regarded as new tracking vehicles. Those tracking vehicles that are not associated with the detection results are considered to have disappeared from the monitoring range of the vehicle and are removed from the tracking results.

S17. Verification: verify the object by tracking its motion and shape information. If a tracked object has not moved for a period of time, it is considered stationary and incorporated into the map information. If the motion and shape information of the tracked object changes irregularly in a short period of time, this result is removed. Only those tracking results with normal changes in motion and shape are considered as candidate vehicles.

As shown in Figure 3, the visual image sample extraction step further includes:

S2. Visual sample image extraction:

In the present invention, a visual sample picture extraction method is disclosed. By using the time series of parameters such as the position and direction of the candidate vehicle relative to the data collection vehicle obtained by S1, according to the position and direction of the candidate vehicle at each moment, according to the laser sensor and image acquisition The geometric relationship between the devices, project the candidate vehicle into the image, generate the region of interest, use the detector to correct the region of interest, for each candidate vehicle, calculate the relative position of the candidate vehicle relative to the camera according to its position, direction and other parameters The angle of view removes image frame samples with similar angles of view, and automatically extracts sample pictures of the candidate vehicle under different angles of view. The visual sample image extraction framework is shown in Figure 3, and each module is introduced below.

S21. Region of interest extraction based on laser data: According to the position and direction of the candidate vehicle at each moment, according to the geometric relationship between the laser sensor and the image acquisition device, project the candidate vehicle into the image, and extract the sensory information containing the candidate vehicle. area of interest;

S22. Correction of the region of interest based on image technology: using an image-based detection method to correct the region of interest and find the vehicle therein;

S23. Vehicle sample image extraction and deduplication: according to the correction result, image frame samples with similar viewing angles are removed, and sample images of vehicles under different viewing angles are automatically extracted.

The above working process will be described in detail below.

S2. Visual image sample extraction:

Using the candidate vehicles obtained from laser data processing, the process of extracting vehicle sample pictures from video image data is shown in Figure 3.

S21. Extraction of regions of interest based on laser data: the candidate vehicles obtained by laser data processing include the position, size, and motion information of the vehicles, and project the candidate vehicles into the image at the corresponding time according to the calibration relationship between the laser scanner and the camera , get the ROI containing the candidate vehicle, and get the pose information of the candidate vehicle in the ROI according to the corresponding motion information. Due to the great difficulty in processing laser data in dynamic traffic environments, regions of interest often contain errors that need to be corrected.

S22. Image-based region-of-interest correction: In the present invention, a Ladybug panoramic camera system integrating 6 cameras is used to monitor the environment around the vehicle, and the resulting panoramic image has certain geometric distortions. The geometric distortion on each region of interest needs to be eliminated in advance, and the distortion can be effectively removed by projecting the pixel points on the region of interest to its cut plane in the spherical coordinate system.

Since the pose of the vehicle in the ROI is known, the ROI can be corrected based on different pose angles. A detector based on HOG features is used to refine the region of interest. The USC multi-angle vehicle dataset is used to train classifiers for vehicles with different angles, and the training data is manually classified. For each category, 200 images are selected as positive samples, and 4 classifiers are trained. Vehicle detection is performed on the region of interest, and the detected vehicle with the highest score and higher than a given threshold is selected as a sample picture.

S23. Vehicle sample extraction and deduplication: During the tracking process of the candidate vehicle, its appearance in the picture changes very slowly. The performance of the same vehicle in the image usually has a large difference after many frames. Moreover, when driving on the road, it often happens that the vehicles are relatively stationary, so that the performance of the vehicle will not change for a long time. It is necessary to filter pictures of vehicles with the same or similar performance. The direction of motion of the vehicle relative to the data collection vehicle and its head orientation are the most influential factors on the performance of candidate vehicles in the picture. For each vehicle, its movement direction α and head orientation β relative to the data collection vehicle at each moment can be calculated. The two angles are discretized into an angle grid of 10°×10°, and the image with the highest score during the correction process is extracted for the same vehicle in each angle grid as a sample. Moreover, the pose of the vehicle in the sample picture is determined according to the difference between the two angles, and the sample picture is divided into 8 categories.

As mentioned above, although the Example of this invention was demonstrated in detail, it is obvious to those skilled in the art that many modifications can be made as long as the inventive point and effect of this invention are not substantially deviated. Therefore, all such modified examples are also included in the protection scope of the present invention.

Claims (8)

1., based on a multi-modal sensing data automatic acquisition vehicle training sample method, it is characterized in that, comprise the following steps:

Vehicle detection step based on laser, locator data: according to the distance of laser data, angle and laser sensor calibrating parameters, obtain the two-dimensional coordinate relative to data acquisition vehicle, to describe the profile information of object level; By the analysis of shape, and the detection of mobile object is followed the trail of, and extracts candidate's vehicle relative to the isoparametric time series of the locality of data acquisition vehicle;

Visual pattern sample extraction step: according to the locality of candidate's vehicle in each moment, according to the geometric relationship between laser sensor and image capture device, this candidate's vehicle is projected in image, produce area-of-interest, and use detecting device to revise area-of-interest, to each candidate's vehicle, the relative perspective of this candidate's vehicle relative to video camera is calculated according to parameters such as its localities, remove the picture frame sample that visual angle is close, automatically extract the samples pictures of this candidate's vehicle under different visual angles;

The described vehicle detection step based on laser, locator data comprises further:

Data fusion: will the identical of each laser scanner be come from or close on the data fusion of time;

Cluster: will come from the data of each laser sensor, according to the distance of adjacent point-to-point transmission, carries out cluster;

Mark: cluster is divided into stationary object, mobile object or uncertain three types; Map generates: the map generating data of description collection vehicle motion track surrounding static environment;

Detect: find candidate's vehicle current carrying out in sorted laser fusion data; Follow the trail of: interactive calculation result and tracking result before, upgrade tracking state and car body, kinematic parameter;

Checking: it is verified by the motion of tracking objects and shape information;

Described visual pattern sample extraction step comprises further:

Region of interesting extraction based on laser data: laser data is followed the trail of the calibrating parameters of result according to camera and laser, project in image, extract the area-of-interest comprising candidate's vehicle;

Area-of-interest correction based on image technique: use the detection method based on image, revise area-of-interest, finds vehicle wherein;

Vehicle sample image extracts and duplicate removal: according to correction result, remove the image that same vehicle pose is identical or close, obtains vehicle sample image through screening.

2. one according to claim 1 is based on multi-modal sensing data automatic acquisition vehicle training sample method, and it is characterized in that, described multi-modal sensor comprises: multi visual angle laser scanner, multi-visual angle filming head and GPS/IMU positioning system; Described multi visual angle laser scanner, multi-visual angle filming head are used for Monitoring Data collection vehicle surrounding environment, form omnibearing covering.

3. one according to claim 2 is based on multi-modal sensing data automatic acquisition vehicle training sample method, it is characterized in that, described GPS/IMU is used for the pose of the 6DOF of measuring vehicle.

4. one according to claim 1 is based on multi-modal sensing data automatic acquisition vehicle training sample method, it is characterized in that, described detecting device adopts the detection method based on image.

5. one according to claim 1 is based on multi-modal sensing data automatic acquisition vehicle training sample method, and it is characterized in that, described tracking is specially: do not find the testing result that can associate and follow the trail of result for those, be considered as new tracking vehicle; And those are not associated with the tracking vehicle of testing result, then think that it disappears in the monitoring range of vehicle, remove from tracking result.

6. one according to claim 1 is based on multi-modal sensing data automatic acquisition vehicle training sample method, it is characterized in that, described checking is specially: if the not motion within a period of time of certain tracking objects, be then considered as static and incorporated in cartographic information; If the motion of tracking objects and shape information there occurs erratic change at short notice, then remove this result; The tracking result of those motions and shape normal variation is only had just to be regarded as candidate's vehicle.

7. one according to claim 1 is based on multi-modal sensing data automatic acquisition vehicle training sample method, it is characterized in that, the correction of described area-of-interest is specially: according to the pose of candidate's vehicle in area-of-interest, selects the detecting device based on specific pose vehicle training to revise it.

8. one according to claim 1 is based on multi-modal sensing data automatic acquisition vehicle training sample method, it is characterized in that, described removal repeated sample is specially: according to vehicle relative to the direction of motion of data acquisition vehicle and headstock towards, apparent identical or close sample image is screened.