CN110717445B - Front vehicle distance tracking system and method for automatic driving - Google Patents

Abstract

The invention discloses a preceding vehicle distance tracking system and method for automatic driving, wherein: a data acquisition unit is used to acquire image sequences with equal time intervals from a camera; a vehicle detection unit is used to extract the outline frame of the preceding vehicle from the image line; the coordinate positioning unit is used to calculate the real position of the vehicle ahead according to the pixel coordinates of the vehicle outline frame line in the image and the camera parameters; the vehicle tracking unit identifies the front vehicle outline frame line of each image in the serial image sequence For the same vehicle, the vehicle is numbered; according to the vehicle position calculated by the coordinate positioning unit and the vehicle number calculated by the vehicle tracking unit, the system converts the input image sequence into an XML format file of the distance sequence of the preceding vehicle. Through the technical solution of the present invention, a single vehicle-mounted camera can be used to detect and track the position of the vehicle ahead on the road, which is beneficial to road perception, automatic obstacle avoidance and decision-making assistance in the automatic driving system.

Description

A front vehicle distance tracking system and method for automatic driving

technical field

The present invention relates to the technical field of visual aided automatic driving, in particular, to a preceding vehicle distance tracking system for automatic driving and a two-dimensional image-based vehicle ranging method.

Background technique

Autonomous driving based on visual assistance is the mainstream solution for current autonomous driving technology. Cameras and lidars are the two most commonly used vehicle vision sensors, among which cameras collect 2D image data and lidars collect 3D point cloud data. Compared with cameras, the current cost of LiDAR is relatively high. Therefore, cameras are the lowest cost solution to achieve vision-assisted autonomous driving.

On the vehicle-mounted camera autonomous driving platform, image-oriented road condition recognition and understanding is the core issue of autonomous driving. The automatic decision-making technologies such as vehicle avoidance and path planning involved in autonomous driving all need to obtain the position information of the vehicle ahead. At the same time, by tracking the vehicles on the road ahead, it can further serve scene understanding and driving decision-making tasks such as predicting the driving behavior of the preceding vehicle and braking in advance. Therefore, the image-based approach and system for distance tracking of the preceding vehicle are the key technologies for autonomous driving.

In the prior art, traditional graphics operators are usually used to extract the information of the vehicle ahead in the image, which cannot be well adapted to complex road conditions. situation is prone to misjudgment. At the same time, most of the existing technologies do not consider the correlation of multiple frames of images, do not match different images of the same vehicle from consecutive multiple frame images, and ignore the continuous tracking of the position of the vehicle ahead, thus limiting the application of the ranging results. scope. Specifically, there are the following difficulties in using the traditional visual ranging method of the preceding vehicle to assist automatic driving decision-making:

1) Complexity: The application scenarios of autonomous driving are very complex. The illumination levels, road textures, and occlusions of images taken under different road conditions are different. Traditional methods lack robustness in complex scenes with strong interference;

2) Representation ability: The existing vehicle ranging method does not integrate vehicle tracking technology, and cannot concatenate multiple instantaneous coordinates of the vehicle ahead into the trajectory of the vehicle ahead, and cannot infer the driving behavior of the vehicle ahead.

SUMMARY OF THE INVENTION

The purpose of the present invention is to track the trajectory of the preceding vehicle and calculate the position of the preceding vehicle based on the image sequence captured by the vehicle-mounted camera, thereby assisting automatic driving decision-making.

The technical solution of the present invention provides a preceding vehicle distance tracking system for automatic driving, wherein the system includes: a data acquisition unit, a vehicle detection unit, a coordinate positioning unit and a vehicle tracking unit;

The data acquisition unit is used to extract image sequences at equal time intervals from the camera, and record the parameters of the camera at the same time, including internal parameters representing camera focal length, projection center, tilt and distortion, and external parameters representing camera position translation and position rotation;

The vehicle detection unit includes several vehicle detection modules, wherein each vehicle detection module is responsible for processing a single image; the vehicle detection module uses the classifier constructed by the deep neural network model to identify the vehicle in the image, and uses the regressor to fit the contour frame of the vehicle. line, so as to locate the vehicle ahead on a single image;

The coordinate positioning unit includes several coordinate positioning modules, wherein each coordinate positioning module is responsible for processing a single vehicle outline frame; for each vehicle outline frame, the coordinate positioning module is based on the pixels of the four vertices of the frame line and the data collected by the data acquisition unit. The obtained camera parameters, through the method of geometric transformation, transform the pixel coordinates of the outline frame line of the front vehicle into the vehicle body coordinates where the vehicle camera is located, so as to determine the spatial position of the front vehicle relative to the vehicle;

The vehicle tracking unit is responsible for processing and recognizing the same vehicle in multiple frames of images to form the driving track of each vehicle; the vehicle tracking unit identifies the same vehicle appearing in different pictures, assigns it a unique id number, and assigns the driving track to the vehicle. The trajectories are connected in series, so that the preceding vehicle can be tracked.

Further, the vehicle detection unit includes a candidate region generation module, a vehicle discrimination module and a contour frame line regression module;

The candidate region generation module saves several anchor points of different sizes, and each anchor point is a rectangular area composed of several pixels;

The input of the vehicle identification module is a candidate region that may contain a vehicle generated by the candidate region generation module, and the output is the information of whether the candidate region contains a vehicle;

The contour box line regression module fine-tunes the contour line of the vehicle in the picture based on the coordinates of the candidate region based on the mask region convolutional neural network regressor.

Further, according to the vehicle detection unit, use the mask area convolutional neural network to calculate the outline frame line of the vehicle in the image; wherein, the calculation of the mask area convolutional neural network is divided into three steps:

Step 1. Extract the candidate area, traverse the entire image from left to right and top to bottom with several preset anchor points of different sizes, and calculate the position of the rectangular block that may be used as the outline of the vehicle;

Step 2. Perform regional object classification. For each candidate region, use a convolutional neural network to extract the visual features of the region, and use a multi-layer perceptron to judge the category of objects in the region through the visual features;

Step 3. Complete the coordinates of the outline frame. For each candidate area, use the neural network to regress the offset of the outline frame of the candidate area relative to the outline frame of the detection target, so that the outline frame of the candidate area is further fitted. Detects the outline of the target.

Further, the coordinate positioning unit is configured to:

According to the coordinates of the vehicle in the image obtained by the vehicle detection unit, the coordinates of the vehicle in the world coordinate system are obtained through the camera coordinate-world coordinate transformation formula, where the coordinate transformation formula is:

sm=A[R|t]M

where M=[x _w , y _w , z _w ] ^T is the three-dimensional coordinate in the world coordinate system, m=[x _i , y _i ] ^T is the two-dimensional coordinate of the bottom center point of the vehicle outline frame line detected in the image , R and t are the rotation matrix and translation matrix in the camera external parameter matrix, respectively; A is the camera internal parameter matrix, _- A[1,1]=f _x , A[1,3]=c _x , A[2, 2]=f _y , A[2,3]= _cy , A[3,3]=1, the rest of A is 0; where f _x , f _y , c _x , _cy are the camera at x The focal length in the axial direction, the focal length in the y-axis direction, the optical center in the x-axis direction, and the optical center in the y-axis direction; s is the depth of field; among them, m, A, R, t are all available from the data The known quantities obtained by the acquisition unit, s, M are the unknown quantities to be obtained.

Further, the calculation of the coordinate positioning unit is divided into 2 steps:

Step 1. Depth of field estimation, take a point on the ground of the horizontal line at the bottom of the vehicle frame line on the image, its two-dimensional coordinate in the image is m _g , the z-direction component of the world coordinate of this point z _w = 0 is known, and because The ground point and the bottom center point of the vehicle frame line are located on the same horizontal plane of the image, and the depth of field s of these two pixel points is the same. Therefore, the first linear equation system (e31) can be obtained.

sm _g =A[R|t]M ₀ (e31)

Step 2, according to the world coordinate solution step, the second linear equation system (e32) is obtained from the center point m at the bottom of the vehicle frame line by the coordinate transformation formula

sm=A[R|t]M (e32)

By combining these two equations, the unknown depth of field s in the equation (e32) can be eliminated with the known in the equation (e31), so as to obtain the world coordinate M of the center point at the bottom of the vehicle frame line.

Further, the vehicle tracking unit includes a distance calculation module and a distance matching module

The distance calculation module calculates the pixel distance from the center of several outline frames of the first frame to several outline frames of the second frame between the two frames of images according to each vehicle outline frame line of the two frames of pictures, and the distance from the nearest set of frames The matching outline frame line is considered to be the outline frame line of the same vehicle in two frames of pictures;

For the two adjacent pictures in the image sequence, the distance matching module preferentially matches the vehicle outline frame line with the closest distance in the adjacent two pictures according to the nearest matching principle, and assigns the matched adjacent group of vehicle outline frame lines to the same. Vehicle ID mark, and then remove the matched vehicle outline frame, and continue to match the remaining outline frames in the two adjacent pictures according to the principle of closest distance, until all the vehicles in one picture in the adjacent two pictures The outline frame lines have all been matched.

The present invention also provides a method for tracking by a preceding vehicle distance tracking system for automatic driving, which specifically includes the following steps:

Step 1: Camera calibration, the position parameters and optical parameters of the vehicle camera are calibrated and recorded in the system data acquisition software;

The camera position parameters include the distance from the fixed position of the camera to the front of the car, the chassis of the car, the sides of the car body, and the stereo angle of the camera relative to the chassis of the car;

Step 2: Identify and locate the vehicle in front, and the identification and location of the vehicle in front are realized by the vehicle detection unit and the coordinate positioning unit;

Step 3: Track the trajectory of the vehicle ahead, identify the vehicles that appear repeatedly in the image sequence through the vehicle tracking and positioning unit, assign different unique IDs to all different vehicles appearing in the image sequence as distinctions, and output the trajectory of each vehicle Sequence to XML file.

The beneficial effect of the present invention is that the robustness of vehicle detection under complex road conditions is improved by using the mask area neural network to identify the vehicle contour in the image. Through the coordinate transformation based on the horizon pixel point, the three-dimensional coordinate point of the vehicle is more accurately restored from the image. Through the vehicle tracking unit, the system not only has the function of distance measurement of the preceding vehicle, but also has the function of vehicle tracking, so that the movement trajectory of the preceding vehicle can be better grasped, and the movement of the preceding vehicle can be predicted, which makes the driving decision of the vehicle more accurate. Anticipation and safety.

Description of drawings

The advantages of the above and/or additional aspects of the present invention will become apparent and readily understood from the following description of embodiments in conjunction with the accompanying drawings, wherein:

FIG. 1 is a schematic block diagram of a method and system for tracking the distance of a preceding vehicle for automatic driving according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a coordinate transformation calculation process according to an embodiment of the present invention.

Detailed ways

In order to understand the above objects, features and advantages of the present invention more clearly, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features of the embodiments may be combined with each other under the condition of no conflict.

In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the following disclosure. Restrictions to specific embodiments.

Example:

Embodiments of the present invention will be described below with reference to FIGS. 1 to 2 .

As shown in FIG. 1 , this embodiment provides a preceding vehicle distance tracking system 100 for automatic driving, including: a data acquisition unit 10 , a vehicle detection unit 20 , a coordinate positioning unit 30 and a vehicle tracking unit 40 .

The data acquisition unit is used to extract image sequences at equal time intervals from the camera, and record the parameters of the camera at the same time, including internal parameters representing camera focal length, projection center, tilt and distortion, and external parameters representing camera position translation and position rotation; In this embodiment, the data acquisition unit 10 includes data acquisition hardware equipment and data acquisition software. The data acquisition hardware is a camera fixed on the top of the vehicle. The lens of the camera is oriented parallel to the chassis of the vehicle. The distance between the camera and the front of the vehicle, the chassis, and the sides of the vehicle body should be measured and recorded in the data acquisition software. When driving, turn on the camera to capture road conditions, and transmit the captured image sequence to subsequent units of the system through data acquisition software.

The data acquisition software is used to transmit the captured road condition image sequence information from the camera and record the parameter information of the camera itself, providing data support for the processing of subsequent units of the system. Specifically, the data acquisition software is divided into an image sequence acquisition module and a camera parameter acquisition module.

The image sequence acquisition module is used to collect the road condition image sequence from the camera and transmit it to the subsequent unit. An in-vehicle front-facing camera captures footage of the road ahead, including road, vehicle and pedestrian conditions. The recorded video will be cut into a sequence of equally spaced images at a fixed frame rate. The image sequence contains multiple frames of pictures, which are represented by "picture 1", "picture 2"..."picture n" in Figure 1, where n represents the total number of pictures in the image sequence. In the image sequence, the pictures follow a time sequence relationship, and the interval between the shooting times of two adjacent pictures is equal.

The camera parameter acquisition module records the external parameters and internal parameters of the camera and transmits them to the subsequent units. Specifically, the external parameters of the camera are the spatial position parameters of the camera placed on the vehicle body, which are stored with a rotation matrix and a translation vector; the external parameters of the camera are the optical parameters of the camera itself, including the focal length and optical center of the camera on the x-axis and y-axis. Components in both directions of the axis.

The vehicle detection unit 20 includes several vehicle detection modules, wherein each vehicle detection module is responsible for processing a single image; the vehicle detection module uses the classifier constructed by the deep neural network model to identify the vehicle in the image, and uses the regressor to fit the contour frame of the vehicle , so as to locate the vehicle ahead on a single image;

In this embodiment, the vehicle detection unit 20 includes: n vehicle detection subunits, which are represented by “vehicle detection unit 21 ”, “vehicle detection unit 22 ” . . . “vehicle detection unit 2n ” in FIG. The total number of pictures in the processed image sequence. Specifically, a vehicle detection subunit is responsible for processing one picture in the image sequence. Each vehicle detection sub-unit has exactly the same configuration, the difference is that the processed images are different.

Specifically, a vehicle detection unit 20 is composed of the following functional modules: a candidate region generation module, a vehicle discrimination module, and a contour frame line regression module.

The candidate region generation module is configured as follows: the candidate region generation module saves several anchor points of different sizes, and each anchor point is a rectangular region composed of several pixels. The candidate region generation module sequentially moves these anchor points from the upper left corner of the picture to the lower right corner of the picture in the order from left to right and from top to bottom, moving one pixel unit each time. At each position where the anchor point moves, the candidate area generation module determines whether the rectangular area covered by the anchor point may contain vehicles according to the pixel characteristics. If the anchor point coverage area may contain vehicles, the position of the anchor point in the picture is recorded as the candidate area. .

The vehicle identification module is configured such that the vehicle identification module inputs a candidate region that may contain a vehicle generated by the candidate region generation module, and outputs information about whether the candidate region contains a vehicle. Further, the vehicle discrimination module performs feature extraction and classification on the pixel features of the candidate region according to the mask region convolutional neural network classifier, and determines whether the candidate region contains a vehicle according to the probability of each category output by the classifier. Specifically, if the probability of the class "vehicle" is the largest in the probability output by the classifier, it is considered that the candidate area contains a vehicle, otherwise, it is considered that the candidate area does not contain a vehicle, and no subsequent processing is required.

The contour box line regression module is configured to fine-tune the contour lines of the vehicle in the image based on the coordinates of the candidate region according to the mask region convolutional neural network regressor. Further, the contour frame regression module is configured to: the contour frame is represented by the left (x, y) of the upper left corner of the frame and the width and height (w, h) of the contour frame. Each piece of fig is converted into a set of outline frame parameters of each vehicle in the picture by the vehicle detection unit: ((x ₁ , y ₁ , w ₁ , h ₁ ), (x ₂ , y ₂ , w ₂ , h ₂ ) ,…,(x _k ,y _k ,w _k ,h _k )). The variable k indicates that the picture fig contains k vehicles, and k can be different values for different pictures in the image sequence.

According to the vehicle detection unit, use the mask area convolutional neural network to calculate the outline frame of the vehicle in the image; wherein, the calculation of the mask area convolutional neural network is divided into three steps:

Step 1. Extract the candidate area, traverse the entire image from left to right and top to bottom with several preset anchor points of different sizes, and calculate the position of the rectangular block that may be used as the outline of the vehicle;

Step 2. Perform regional object classification. For each candidate region, use a convolutional neural network to extract the visual features of the region, and use a multi-layer perceptron to judge the category of objects in the region through the visual features;

Step 3. Complete the coordinates of the outline frame. For each candidate area, use the neural network to regress the offset of the outline frame of the candidate area relative to the outline frame of the detection target, so that the outline frame of the candidate area is further fitted. Detects the outline of the target.

The coordinate positioning unit 30 includes several coordinate positioning modules, wherein each coordinate positioning module is responsible for processing a single vehicle outline frame line; for each vehicle outline frame line, the coordinate positioning module is based on the pixels of the four vertices of the frame line and the data collected by the data acquisition unit ( 10) The obtained camera parameters, through the method of geometric transformation, transform the pixel coordinates of the outline frame line of the preceding vehicle into the vehicle body coordinates where the on-board camera is located, thereby determining the spatial position of the preceding vehicle relative to the vehicle;

In this embodiment, the coordinate positioning unit 30 includes: n coordinate positioning subunits, which are denoted by “coordinate positioning unit 31”, “coordinate positioning unit 32”… “coordinate positioning unit 3n” in FIG. The total number of pictures in the processed image sequence. Specifically, a coordinate positioning sub-unit is responsible for processing a set of outline frame line parameters obtained by a picture in the image sequence through the vehicle detection unit. Each coordinate positioning subunit has exactly the same configuration, the difference is that the set of parameters of the outline frame line processed is different.

Specifically, the coordinate positioning unit is configured to: obtain the coordinates of the vehicle in the world coordinate system through the camera coordinate-world coordinate transformation formula according to the coordinates of the vehicle outline frame line in the image obtained by the vehicle detection unit, wherein the coordinate transformation formula for:

sm=A[R|t]M

where M=[x _w , y _w , z _w ] ^T is the three-dimensional coordinate in the world coordinate system, m=[x _i , y _i ] ^T is the two-dimensional coordinate of the point in the picture taken by the camera, R and t They are the rotation matrix and the translation matrix in the camera extrinsic parameter matrix, respectively. A is the parameter matrix in the camera, A[1,1]=f _x , A[1,3]=c _x , A[2,2]=f _y , A[2,3]= _cy , A[3 ,3]=1, the rest of the positions of A are 0. Where f _x , f _y , c _x , and c _y are the focal length of the camera in the x-axis direction, the focal length in the y-axis direction, and the x-axis direction. The optical center of , the optical center in the y-axis direction. s is the depth of field.

According to the above formula, the calculation of the coordinate positioning unit is divided into two steps: a depth of field estimation step and a world coordinate solution step. In the depth of field estimation step, a reference point is taken on the ground of the horizontal line at the bottom of the vehicle frame line on the image, its two-dimensional coordinate in the image is m _g , and the z-direction component z _w =0 of the world coordinate of this point is known, At the same time, since the ground point and the bottom center point of the vehicle frame line are located on the same horizontal plane of the image, the depth of field s of these two pixel points is the same. From this, the first system of linear equations (e31) can be obtained

sm _g =A[A|t]M ₀

where M ₀ =[x ₀ ,y ₀ ,0] ^T is the three-dimensional coordinate of the reference point in the world coordinate system, m _g =[x _g ,y _g ] ^T is the two-dimensional coordinate of the reference point in the image, R and t They are the rotation matrix and the translation matrix in the camera extrinsic parameter matrix, respectively. A is the parameter matrix in the camera, A[1,1]=f _x , A[1,3]=c _x , A[2,2]=f _y , A[2,3]= _cy , A[3 ,3]=1, the rest of the positions of A are 0. Where f _x , f _y , c _x , and c _y are the focal length of the camera in the x-axis direction, the focal length in the y-axis direction, and the x-axis direction. The optical center of , the optical center in the y-axis direction. s is the depth of field.

In the step of solving the world coordinates, the second linear equation system (e32) is obtained from the center point m at the bottom of the vehicle frame line by the coordinate transformation formula.

sm=A[R|t]M

Among them, M=[x _w , y _w , z _w ] ^T is the three-dimensional coordinate of the bottom center of the vehicle outline frame line detected in the image in the world coordinate system, m=[x _i , y _i ] ^T is the detected image in the image. The two-dimensional coordinates of the bottom center point of the vehicle outline frame line, R and t are the rotation matrix and translation matrix in the camera extrinsic parameter matrix, respectively. A is the parameter matrix in the camera, A[1,1]=f _x , A[1,3]=c _x , A[2,2]=f _y , A[2,3]= _cy , A[3 ,3]=1, the rest of the positions of A are 0. Where f _x , f _y , c _x , and c _y are the focal length of the camera in the x-axis direction, the focal length in the y-axis direction, and the x-axis direction. The optical center of , the optical center in the y-axis direction. s is the depth of field.

By combining these two equations, the unknown depth of field s in the equations (e32) can be eliminated with the knowns in the equations (e31), so as to obtain the world coordinate M of the center point at the bottom of the vehicle frame line.

Further, for each vehicle outline frame line (x, y, w, h) in each image, the coordinates of the center point (x+w/2, y+h) at the bottom of the vehicle outline are calculated, that is, m= (x+w/2, y+h). Since m ₀ and m are on the same horizontal line, correspondingly m ₀ =(x′, y+h). From this, the quantities m, m ₀ , A, R are known , t are all given, the simultaneous equations (e31, e32) can solve the world coordinates (X, Y, Z) of the bottom of the rear edge of the vehicle body. In this way, the coordinates of each vehicle outline frame (x, y, w, h) Convert to world coordinates (X, Y, 0) of the bottom center point of the rear edge of the vehicle body.

The vehicle tracking unit 40 is responsible for processing and recognizing the same vehicle in the multi-frame images to form the driving track of each vehicle; the vehicle tracking unit 40 identifies the same vehicle, assigns it a unique id number, and connects the driving track in series, thereby realizing Front car tracking.

Calculate the pixel distance from the center of several contour lines of the first frame to several contour lines of the second frame between the two frames of images according to the respective vehicle contour lines of the two frames of images, and match the contour between the nearest set of frames. The frame line is considered to be the contour frame line of the same vehicle in two frames of images; thus, by calculating the adjacent images of the entire image sequence, all matching contour frames between frames can be obtained, corresponding to all different frames in the image sequence. Vehicles, for each vehicle, concatenate their outline frame lines in each frame to get the driving trajectory of each vehicle.

In this embodiment, the vehicle tracking unit 40 includes: a distance calculation module and a distance matching module.

Specifically, the distance calculation module calculates the pixel distances from the centers of several outline outlines of the first frame to several outline outlines of the second frame between the two frames of images according to the respective vehicle outline outlines of the two frames of pictures before and after, and the distance to the nearest A set of matching contour lines between frames is considered to be the contour lines of the same vehicle in two frames. For each pair of adjacent pictures in the entire image sequence, the distance is calculated to obtain the distances of all the corresponding vehicle contour lines in the adjacent two pictures.

Specifically, the distance matching module is configured to: for two adjacent pictures in the image sequence, according to the nearest matching principle, preferentially match the vehicle outline frame line with the closest distance in the two adjacent pictures, and match the obtained adjacent group of The vehicle outline is given the same vehicle ID mark, and then the matched vehicle outline is removed, and the remaining outlines in the two adjacent pictures are matched according to the principle of closest distance, until the two adjacent pictures are All the vehicle outlines in one image have been matched, and the remaining vehicle outlines in the other image are ignored. By performing the above matching operation on all adjacent image pairs in the entire image sequence, all matching contour frames between frames and the unique ID of the vehicle corresponding to the contour frame can be obtained. For each vehicle that is given a unique ID, concatenate their outlines in each frame to get the driving trajectory of each vehicle.

Further, the distance calculation module is configured to: for two adjacent images of the same image sequence, there are respectively a sequence of vehicle contours:

and

Among them, k and k' refer to the number of vehicle outline frame lines calculated by the vehicle detection unit 20 in the images fig ₁ and fig ₂ respectively. define distance

可以求出k×k′组车辆轮廓框线之间的距离。

The distances between the k×k′ groups of vehicle contour lines can be obtained.

按照最近的原则找到图像fig₂中的轮廓框线

匹配。依次按顺序计算一个图像序列中所有两两相邻的图像的匹配，可以得到若干连续的匹配串

其中K为图像序列中车辆i出现的次数。综合坐标定位单元30的结果，可以得到车辆i的坐标序列

坐标序列以XML格式输出，作为该前车距离跟踪系统的最终输出结果。Further, the distance matching module is configured as: for each vehicle outline frame line in the image fig ₁

Find the outline frame lines in the image fig ₂ according to the nearest principle

match. Calculate the matching of all pairs of adjacent images in an image sequence in order, and several consecutive matching strings can be obtained

where K is the number of times the vehicle i appears in the image sequence. By synthesizing the results of the coordinate positioning unit 30, the coordinate sequence of the vehicle i can be obtained

The coordinate sequence is output in XML format as the final output result of the preceding vehicle distance tracking system.

This embodiment also provides a method for tracking the distance of a preceding vehicle for automatic driving, which specifically includes the following steps:

Step 1: Camera calibration, the camera calibration step requires that the position parameters and optical parameters of the vehicle camera be calibrated and recorded in the system data acquisition software.

Specifically, the camera position parameters include the distance from the fixed position of the camera to the front of the vehicle, the chassis of the vehicle, and the sides of the vehicle body, and the stereo angle of the camera relative to the chassis of the vehicle. The distance of the camera relative to the front of the car, the chassis, and both sides of the body is represented by the translation matrix t in the camera extrinsic parameter matrix, and the solid angle of the camera relative to the chassis is represented by the rotation matrix R in the camera extrinsic parameter matrix.

The optical parameters of the camera are represented by the camera internal parameter matrix A, where A[1,1]=f _x , A[1,3]=c _x , A[2,2]=f _y , A[2,3]= c _y , A[3,3]=1, the rest of the positions of A are 0. Among them f _x , f _y , c _x , c _y are the focal length of the camera in the x-axis direction, and the focal length in the y-axis direction. , the optical center in the x-axis direction, and the optical center in the y-axis direction.

Step 2: Identifying and Locating the Front Vehicle The recognition and localization of the front vehicle is realized by the vehicle detection unit 20 and the coordinate positioning unit 30 .

Specifically, for a set of image sequences, it includes n pictures with equal time intervals: "picture 1", "picture 2"..."picture n", and the n vehicle detection subunits correspond to each frame of picture in the image sequence. A correspondence. Each vehicle detection subunit detects the outline frame lines of all vehicles in a picture, and each outline frame line is represented by the coordinates of the upper left corner of the outline and the width and height of the outline. For a picture containing k vehicles, the front vehicle identification and localization steps will result in the following set of k groups of contour lines:

((x ₁ ,y ₁ ,w ₁ ,h ₁ ),(x ₂ ,y ₂ ,w ₂ ,h ₂ ),…,(x _k ,y _k ,w _k ,h _k )), where x,y , w, h represent the abscissa of the upper left corner of the outline frame, the ordinate of the upper left corner of the pixel, the width and the height, respectively. The subscripts 1, 2, ..., k correspond to the 1, 2, ..., k vehicles in the picture, respectively.

For each contour frame in each frame of picture, the coordinate positioning unit first obtains the two-dimensional coordinate m=(x) of the midpoint of the vehicle bottom according to the frame parameters (x, y, w, h) of the vehicle contour frame on the picture +w/2,y+h/2). Further, the two-dimensional coordinates of the midpoint of the bottom of the vehicle are converted into three-dimensional coordinates in the real world coordinate system by the coordinate positioning unit M=(X, Y, 0). Wherein X , Y are the lateral distance and longitudinal distance of the vehicle ahead relative to the vehicle in the real world coordinate system, respectively.

Step 3: Track the trajectory of the vehicle ahead. Through the vehicle tracking and positioning unit, the vehicles that appear repeatedly in the image sequence are identified, and all different vehicles appearing in the image sequence are assigned different unique IDs as distinctions, and the trajectory sequence of each vehicle is output to an XML file.

的坐标序列，X,Y分别代表前方车辆相对于本车的横向距离和纵向距离，下标t,t+1,……,T代表该车在摄像机中出现的连续时间序列，上标i表示该车的唯一ID.所有车辆轨迹最终保存在XML文件中，存储格式的示例如下：Through step 2, each frame of picture in the image sequence is transformed into several coordinate points, which respectively represent the position of the vehicle ahead in this frame of picture. The vehicle tracking and positioning unit identifies the same vehicle in two adjacent frames, thereby organizing the vehicle coordinates in different pictures of the entire image sequence into several continuous tracks, each track corresponding to a vehicle ahead with a unique ID. A trajectory is like

The coordinate sequence of , X, Y respectively represent the lateral distance and longitudinal distance of the vehicle ahead relative to the vehicle, the subscript t, t+1, ..., T represents the continuous time sequence of the vehicle appearing in the camera, and the superscript i represents The unique ID of the car. All vehicle trajectories are finally saved in an XML file, an example of the storage format is as follows:

The steps in this application can be adjusted, combined and deleted in sequence according to actual needs.

The units in the device of the present application can be combined, divided and deleted according to actual needs.

Although the present application has been disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The protection scope of the present application is defined by the appended claims, and may include various modifications, alterations and equivalent solutions for the invention without departing from the protection scope and spirit of the present application.

Claims (4)

1. A preceding vehicle distance tracking system for automatic driving, characterized in that the system comprises: a data acquisition unit (10), a vehicle detection unit (20), a coordinate positioning unit (30) and a vehicle tracking unit (40) ); The data acquisition unit (10) is used for extracting image sequences at equal time intervals from the camera, and simultaneously recording the parameters of the camera, including the internal parameters representing the camera focal length, projection center, tilt and distortion, and the camera position translation and position rotation. external parameters; The vehicle detection unit (20) includes several vehicle detection modules, wherein each vehicle detection module is responsible for processing a single image; the vehicle detection module uses a classifier constructed by a deep neural network model to identify the vehicle in the image, and uses a regressor to fit the vehicle , so as to locate the vehicle ahead on a single image; The coordinate positioning unit (30) includes several coordinate positioning modules, wherein each coordinate positioning module is responsible for processing a single vehicle outline frame; The camera parameters obtained by the data acquisition unit (10), through the method of geometric transformation, transform the pixel coordinates of the outline frame line of the preceding vehicle into the vehicle body coordinates where the vehicle camera is located, thereby determining the spatial position of the preceding vehicle relative to the vehicle; The coordinate positioning unit (30) is configured to: According to the coordinates of the vehicle in the image obtained by the vehicle detection unit (20), the coordinates of the vehicle in the world coordinate system are obtained through the camera coordinate-world coordinate transformation formula, where the coordinate transformation formula is:

in

is the three-dimensional coordinate in the world coordinate system,

is the two-dimensional coordinate of the bottom center point of the vehicle outline frame line detected in the image,

and

are the rotation matrix and translation matrix in the camera extrinsic parameter matrix, respectively

is the in-camera parameter matrix,

,

The rest of the positions are 0; where are the focal length of the camera in the x-axis direction, the focal length in the y-axis direction, the optical center in the x-axis direction, and the optical center in the y-axis direction

is the depth of field; where,

are all known quantities that can be obtained from the data acquisition unit (10),

is the unknown quantity to be sought; The calculation of the coordinate positioning unit is divided into 2 steps: Step 1. Depth of field estimation, take a point on the ground of the horizontal line at the bottom of the vehicle frame line on the image, and its two-dimensional coordinates in the image are

, the z-direction component of the point's world coordinates

It is known that since the ground point and the center point at the bottom of the vehicle frame line are located on the same level of the image, the depth of field of these two pixel points is

The same, from this, the first linear equation system (e31) can be obtained

(e31) Step 2: According to the world coordinate solution step, the center point at the bottom of the vehicle frame line is determined by the coordinate transformation formula.

get the second system of linear equations (e32)

(e32) By combining these two equations, the unknown depth of field in equations (e32) can be eliminated by the knowns in equations (e31)

, so as to find the world coordinates of the center point at the bottom of the vehicle frame line

; The vehicle tracking unit (40) is responsible for processing and recognizing the same vehicle in multiple frames of images to form the driving track of each vehicle; the vehicle tracking unit (40) identifies the same vehicle that appears in different pictures, and assigns it a unique vehicle. id number, and connect the driving trajectories in series, so as to realize the tracking of the preceding vehicle; The vehicle tracking unit (40) includes a distance calculation module and a distance matching module; The distance calculation module calculates the pixel distance from the center of several outline outlines of the first frame to several outline outlines of the second frame between the two frame images according to each vehicle outline frame line of the two frames of images, and the distance to the nearest set of frames The matching outline frame line is considered to be the outline frame line of the same vehicle in two frames of pictures; For the two adjacent images in the image sequence, the distance matching module preferentially matches the vehicle outline frame lines with the closest distance in the adjacent two images according to the nearest matching principle, and assigns the matched adjacent group of vehicle outline frames to the same value. The vehicle ID mark, and then remove the matched vehicle outline, and continue to match the remaining vehicle outlines in the two adjacent images according to the principle of the closest distance, until all the adjacent two images in one image. The vehicle outline lines have all been matched. 2. The preceding vehicle distance tracking system for automatic driving according to claim 1, wherein the vehicle detection unit (20) comprises a candidate region generation module, a vehicle discrimination module and a contour box line regression module; The candidate region generation module saves several anchor points of different sizes, and each anchor point is a rectangular area composed of several pixels; The input of the vehicle identification module is a candidate region that may contain a vehicle generated by the candidate region generation module, and the output is the information of whether the candidate region contains a vehicle; The contour box line regression module fine-tunes the contour line of the vehicle in the image based on the coordinates of the candidate region based on the mask region convolutional neural network regressor. 3. The preceding vehicle distance tracking system for automatic driving as claimed in claim 2, characterized in that, According to the vehicle detection unit, use the mask area convolutional neural network to calculate the outline frame of the vehicle in the image; wherein, the calculation of the mask area convolutional neural network is divided into three steps: Step 1. Extract the candidate area, traverse the entire image from left to right and top to bottom with several preset anchor points of different sizes, and calculate the position of the rectangular block that may be used as the outline of the vehicle; Step 2. Perform regional object classification. For each candidate region, use a convolutional neural network to extract the visual features of the region, and use a multi-layer perceptron to judge the category of objects in the region through the visual features; Step 3. Complete the coordinates of the outline frame. For each candidate area, use the neural network to regress the offset of the outline frame of the candidate area relative to the outline frame of the detection target, so that the outline frame of the candidate area is further fitted. Detects the outline of the target. 4. a method utilizing the preceding vehicle distance tracking system for automatic driving according to claim 1 to track, specifically comprises the following steps: Step 1: Camera calibration, the position parameters and optical parameters of the vehicle camera are calibrated and recorded in the system data acquisition software; The camera position parameters include the distance from the fixed position of the camera to the front of the car, the chassis of the car, the sides of the car body, and the stereo angle of the camera relative to the chassis of the car; Step 2: Identify and locate the vehicle ahead, and the identification and positioning of the vehicle in front is realized by the vehicle detection unit (20) and the coordinate positioning unit (30); Step 3: Track the trajectory of the vehicle ahead, identify the vehicles that appear repeatedly in the image sequence through the vehicle tracking and positioning unit, assign different unique IDs to all different vehicles appearing in the image sequence as distinctions, and output the trajectory of each vehicle Sequence to XML file.

Images