CN116168088B - Monocular side angle imaging vehicle scale estimation and speed estimation method - Google Patents

Abstract

A vehicle scale estimation and speed estimation method for single-target side angle imaging, the steps are as follows: S1, establish a target detection model, detect vehicle targets, wheels, key points, and lane lines; S2, perform target tracking, obtain the tracking sequence of each vehicle, the vehicle detection frame, and the vehicle classification result; S3, calculate the coordinates of the wheel center point "projected" to the road surface to obtain the axle information and divide the lane lines to obtain multiple lanes; S4, based on the tracking sequence of similar targets in step S2 and the axle information of similar targets in step S3, self-calibrate the corresponding targets, and obtain the mapping relationship between the pixel coordinate system and the actual coordinate system of the target; S5, determine the target vehicle, obtain its tracking sequence, axle information at different positions, and coordinate information, and based on the mapping relationship in step S4, obtain the estimated value of the final length of the target vehicle and the estimated value of the vehicle speed. The present invention can automatically calibrate and reduce the complexity of manual operation.

Description

A vehicle scale estimation and speed estimation method based on single-target side angle imaging

Technical Field

The invention belongs to the field of traffic technology, and in particular relates to a vehicle scale estimation and speed estimation method based on single-object side angle imaging.

Background Art

Vehicle length, width and speed estimation is an important topic in the field of transportation. Commonly used methods include 3D solutions, 3D/2D combined solutions, pure 2D solutions, etc. Solutions that introduce 3D sensing usually have high hardware costs. Using pure 2D solutions usually requires manual calibration. Every time a new scene is installed or the camera posture is adjusted, it needs to be recalibrated, which is a lot of work.

Summary of the invention

In view of the problems existing in the above background technology introduction, the purpose of the present invention is to provide a vehicle scale estimation and speed estimation method for single-purpose side angle imaging that can be automatically calibrated, has high accuracy, and reduces the complexity of manual operation.

The technical solution adopted by the present invention is:

A vehicle scale estimation and speed estimation method based on single-object side angle imaging, the specific steps of which are as follows:

S1, establish a target detection model to detect vehicle targets, wheels, key points, and lane lines;

S2, perform target tracking, obtain the tracking sequence of each vehicle, as well as the vehicle detection frame and vehicle classification results;

S3, calculating the coordinates of the wheel center point "projected" onto the road surface to obtain axle information and segmenting the lane lines to obtain multiple lanes;

S4, based on the tracking sequence of the same type of target in step S2 and the axle information of the same type of target in step S3, self-calibrate the corresponding target to obtain the mapping relationship between the pixel coordinate system and the actual coordinate system of the target;

S5, determine the target vehicle, obtain its tracking sequence, axle information at different positions, and coordinate information, and based on the mapping relationship in step S4, obtain the estimated value of the final length of the target vehicle and the estimated value of the vehicle speed.

Furthermore, the target detection model in step S1 is obtained by expanding the hybrid head model based on the yolov5 target detector, which includes a backbone network, and the backbone network is followed by four output branches, each of which corresponds to an upsampling pyramid structure. Output branch 1 outputs an axis-aligned target frame and a vehicle category for vehicle target detection, and output branch 2 outputs an ellipse for wheel detection; output branch 3 outputs key points for locating key points 1, 2, and 3 of the vehicle circumscribed rectangle projected onto the ground, wherein the line connecting key points 1 and 2 is along the direction of the lane line, and the line connecting key points 2 and 3 is perpendicular to the direction of the lane line; output branch 4 outputs a semantic segmentation map for detecting lane lines.

Furthermore, each frame of output branch 1, output branch 2, and output branch 3 is predicted, and output branch 4 is predicted at a set interval.

Furthermore, the acquired data set is divided into a lane line data set and a vehicle data set, and labeled separately.

Further, the tracking sequence in step S2 is obtained based on the iou tracking mode and the output result of the output branch one.

Further, the specific steps of calculating the coordinates of the wheel center point "projected onto" the road surface in step S3 include:

S31, obtaining straight line A based on the connection line between key point 1 and key point 2;

S32, calculating the coordinates of the perpendicular point from the center point of the wheel to the straight line A, that is, the coordinates of the projection point of the wheel center.

Further, the specific steps of the self-calibration in step S4 are as follows:

S41, coordinate system setting. Each lane is defined by two lane lines. The direction "along the outer lane line" is the first axis. The direction "perpendicular to the outer lane line" is the second axis. The intersection of the "outer lane line" and the right edge of the image is the origin.

The mapping formula between camera coordinates and actual space coordinates is as follows:

Where L represents the distance from the camera focus to the actual measurement point; L_real represents the actual length of a point in space; L_pixel represents the pixel length of this point after taking a photo; f represents the focal length of the camera; cellSize represents the pixel size; f and cellSize are fixed values;

The calculation formula of L is as follows:

Among them, H0 represents the "vertical distance between the camera focus and axis one", H1 represents the "coordinate value of the measuring point in the direction of axis two", R0 represents the "vertical distance between the camera focus and axis two", and R1 represents the "coordinate value of the measuring point in the direction of axis one"; once the camera is fixedly installed, H0 and R0 become fixed values.

Substituting formula 2 into formula 1, we get the new expression:

Ratio represents the conversion ratio from pixel coordinates to actual coordinates. As can be seen from formula 3:

Ratio＝f(H1,R1) (Formula 4)

Since H0, R0, f, and cell_size are fixed values, Ratio can be expressed as a function controlled by the H1 and R1 variables;

In order to simplify the calculation, f(H1, R1) is converted into a plane function. The simplified specific formula is as follows:

Ratio＝a*H1+b*R1+c (Formula 5)

Among them, a/b/c are the parameters to be found;

S42, using the least squares method to fit, requires the introduction of multiple sets of known points

{(H1_0,R1_0,Ratio_0),(H1_1,R1_1,Ratio_1),......,

(H1_n, R1_n, Ratio_n)}, referred to as the fitting point set; each point is composed of a triplet, including the pixel coordinates of the point in the second direction of the axis, the pixel coordinates of the point in the first direction of the axis, and the scale of the point;

S43, determining the triplet, and obtaining the pixel coordinates of the axis of each vehicle target through model detection and post-processing logic;

According to formula 3, the actual wheelbase can be expressed as follows:

Among them, L _{real wheelbase} is the actual wheelbase of the vehicle; L _{pixel wheelbase} is the pixel wheelbase of the vehicle; d _x is the differential unit;

Approximate calculation of formula 7:

Since Formula 7 and Formula 8 are very close, the error can be ignored, so Formula 8 is subsequently decomposed to obtain the calculation formula of the triplet:

Among them, H1, R1, L _{pixel wheelbase} -----pixel-level coordinate data can be calculated based on model output + post-processing; L _{real wheelbase is} 2500mm;

S44, according to the triplet calculation method of step S43, based on the 200 sets of car tracking sequences for calibration reference, multiple triplet groups are calculated to obtain the fitting point set of step S42;

S45, fitting,

a. The first fitting is based on all the triplets, fitting the plane, and calculating a set of (a, b, c); assuming that the parameters after calibration are (a, b, c) = (a1, b1, c1); substituting the above structure into formula 5. We get:

Ratio＝a1*H1+b1*R1+c1 (Formula 11)

b. Eliminate triplets with large deviations

Based on Formula 11 and Formula 9; the calculated deviation is:

Where pia represents the deviation of the scale calculated by formula 11 from the scale in formula 9; the deviation of all triples calculated based on formula 12; the top 20% of the triplets with the largest deviations are deleted; and 80% of the triplets are retained;

c. The second fitting, for the remaining triples in b, fit the plane and calculate a set (a, b, c); assume that the parameters after calibration are (a, b, c) = (a2, b2, c2). Substitute the above structure into formula 5; get:

Ratio＝a2*H1+b2*R1+c2 (Formula 12)

At this point, self-calibration is completed.

Further, the calculation steps of the target vehicle final length estimation value in step S5 are as follows:

S51, obtaining a tracking sequence of the target vehicle, and calculating the “pixel-level length” based on “key point 1, key point 2” of the vehicle at different positions in the sequence;

S52, based on formula 12, converting the length at the pixel level into the spatial length;

S53, averaging multiple spatial lengths in the sequence to obtain an estimated final length of the vehicle.

Further, the calculation steps of the target vehicle speed calculation in step S5 are as follows:

S54, obtaining a tracking sequence of the target vehicle, and calculating the projection coordinates of the center point of the front axle of the vehicle to the ground based on the axle center point results of the vehicle at different positions in the sequence;

S55, calculating the distance at the pixel level based on the “ground projection coordinates of the center point of the front axle” of the vehicle at the “first position” and the “ground projection coordinates of the center point of the front axle” of the vehicle at the “last position”;

S56, performing coordinate conversion based on formula 12 to obtain the actual distance between the “first position” and the “last position”;

S57, calculating the time difference between "the imaging time of the frame where the first position is located" and "the imaging time of the frame where the last position is located";

S58, based on the actual distance/imaging time difference between the front and rear ends, calculate the estimated value of the vehicle speed.

Furthermore, in step S5, the vehicle width and the number of vehicle axles can also be calculated;

The steps to calculate the vehicle width are as follows:

Use "Key Point 2/Key Point 3" to calculate the vehicle pixel width;

Calculate the coordinates of the midpoint of the line connecting key point 2 and key point 3, and further calculate the intersection of the "vertical line from the center point to the left lane line" and the two lane lines; the pixel distance between the two intersection points represents the lane pixel width;

The vehicle width is calculated using the following formula: (vehicle pixel width/lane pixel width)*preconfigured lane line width;

The steps to calculate the number of vehicle axles are as follows:

Get the tracking sequence of the target vehicle based on the number of axles of the vehicle at different positions in the sequence;

Using the multi-frame voting method, if the number of occurrences of a certain "axle number" is greater than half of the total number of tracking sequences, and the "axle number" is greater than or equal to 2, it will be taken as the final result; otherwise, no axle detection result will be returned.

Compared with the prior art, the present invention has the following significant advantages: automatic calibration, high precision, and reduced complexity of manual operation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic structural diagram of a mixing head model of the present invention;

FIG2 is a result schematic diagram of a vehicle target frame of the present invention;

FIG3 is a schematic diagram of wheel detection results of the present invention;

FIG4 is a schematic diagram of the results of the key points of the present invention;

FIG5 is a schematic diagram of lane lines/lanes of the present invention;

FIG6 is a schematic diagram of a tracking sequence of the same vehicle of the present invention;

FIG7 is a schematic diagram of a wheel center projection point of the present invention;

FIG8 is a schematic diagram of lane lines/lanes of the present invention;

FIG9 is a schematic diagram of establishing a coordinate system of the present invention;

FIG10 is an exploded schematic diagram of the length L of the present invention;

FIG11 is a schematic diagram of scales of different coordinates of the present invention;

FIG12 is a schematic diagram of sequential imaging of a single vehicle of the present invention;

FIG13 is a schematic diagram of the distance between a vehicle and a focus of the present invention;

FIG. 14 is a schematic diagram of the center points of the front and rear wheels of the present invention.

FIG. 15 is a schematic diagram of a triplet point set of the present invention.

DETAILED DESCRIPTION

The present invention is further described below in conjunction with specific embodiments, but the present invention is not limited to these specific embodiments. Those skilled in the art should recognize that the present invention covers all possible alternatives, improvements and equivalents within the scope of the claims.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "clockwise", "counterclockwise" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more, unless otherwise clearly defined.

In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

Explanation of professional terms:

Yolov5 is a single-stage target detection algorithm. It adds some new improvement ideas based on YOLOv4, which greatly improves its speed and accuracy. The main improvement ideas are as follows: Input: In the model training stage, some improvement ideas are proposed, mainly including Mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling; Baseline network: Integrate some new ideas in other detection algorithms, mainly including: Focus structure and CSP structure; Neck network: The target detection network often inserts some layers between BackBone and the final Head output layer. Yolov5 adds FPN+PAN structure; Head output layer: The anchor box mechanism of the output layer is the same as YOLOv4. The main improvements are the loss function GIOU_Loss during training and the prediction box screening DIOU_nms.

The iou tracking method is a target tracking method.

This embodiment provides a vehicle size estimation and speed estimation method based on single-object side angle imaging, and the specific steps are as follows:

S1, establish a target detection model to detect vehicle targets, wheels, key points, and lane lines;

The target detection model is obtained by extending the hybrid head model based on the yolov5 target detector, which includes a backbone network, and the backbone network is followed by four output branches, each of which corresponds to an upsampling pyramid structure, as shown in FIG1 .

The output of output branch 1 is the axis-aligned target box and vehicle category, which is used for vehicle target detection. The original upsampling pyramid structure is used, with a total of 3 layers. The scaling ratio of each layer relative to the input image is 1/32, 1/16, and 1/8 respectively. The 3 layers in the upsampling pyramid are used as the output layers of branch 1. Vehicle categories include: sedan, SUV, bread, business, bus, small truck, large truck, and others. In the post-processing method, the "maximum suppression" commonly used in the detection model is directly used to obtain the final output box. See Figure 2.

The output of the second output branch is an ellipse, which is used for wheel detection. The corresponding upsampling pyramid structure has 3 layers (1/32, 1/16, 1/8), and only 1/8 layer is used as the output layer. The post-processing method directly uses maximum suppression, which is similar to the maximum suppression of the rectangular frame, to obtain the final output ellipse. See Figure 3.

Output branch 3 outputs key points, which are used to locate key points 1, 2, and 3 of the vehicle's circumscribed rectangle projected onto the ground. The line connecting key points 1 and 2 is along the lane line, and the line connecting key points 2 and 3 is perpendicular to the lane line. The corresponding upsampling pyramid structure has 3 layers (1/32, 1/16, 1/8, 1/4), and only 1/4 layer is used as heatmap. Post-processing uses the peak filtering method of the heatmap-like key point layer to suppress the maximum value and obtain the final output key points. See Figure 4.

The output of output branch 4 is a semantic segmentation map, which is used to detect lane lines. The corresponding upsampling pyramid structure has 3 layers (1/32, 1/16, 1/8, 1/4), and only 1/4 layer is used as the semantic segmentation output layer. The post-processing method is to first perform threshold segmentation on the output layer to obtain a binary map; then perform skeleton extraction on the binary map to obtain a skeleton map; finally, perform Hough transform fitting on the skeleton to obtain the linear coordinates of the lane line. See Figure 5.

Backbone network: Inference is performed for each frame. Output branches 1, 2, and 3 are predicted for each frame to obtain vehicle detection results, key points, and wheel detection results for each frame to maintain the continuity of vehicle results. Output branch 4 is predicted at set intervals (for example, once every 50 frames) to improve the overall inference speed. Lane lines are mainly used to assist calibration and do not need to be detected in every frame.

When training the model, it is important to note that the acquired data set is divided into a lane line data set and a vehicle data set, and they are labeled separately. The lane line data set only has lane line labels but no vehicle labels; the vehicle data set only has vehicle labels but no lane line labels. Lane line samples only supervise branch 4, and branches 1/2/3 are not involved in the feedback; vehicle samples only supervise branches 1/2/3, and do not supervise branch 4.

S2, perform target tracking, obtain the tracking sequence of each vehicle, the vehicle detection frame and the vehicle classification result; the tracking sequence is obtained based on the IOU tracking method and the output result of output branch 1. The output result of branch 1 includes the coordinates of the vehicle detection frame and the classification result. See Figure 6.

S3, calculating the coordinates of the wheel center point "projected" onto the road surface to obtain axle information and segmenting the lane lines to obtain multiple lanes;

The specific steps of calculating the coordinates of the wheel center point "projected" to the road surface include:

S31, obtaining straight line A based on the connection line between key point 1 and key point 2;

S32, calculating the coordinates of the perpendicular point from the wheel center point to the straight line A, that is, the coordinates of the projection point of the wheel center, as shown in FIG7 .

Based on the lane line detection results in step S1, multiple lanes are segmented. Every two lane lines can represent a lane area. For easy distinction, the two lane lines in the same lane are called "near lane line" and "far lane line". As shown in Figure 8, the reason why each lane is illustrated separately is that the subsequent calibration and calculation are all performed according to the lane. Each lane corresponds to a set of calibration parameters.

S4, based on the tracking sequence of the same type of target in step S2 and the axle information of the same type of target in step S3, self-calibrate the corresponding target to obtain the mapping relationship between the pixel coordinate system and the actual coordinate system of the target;

The specific steps of the self-calibration in step S4 are as follows:

S41, coordinate system setting, as shown in FIG9 , each lane is defined by two lane lines, with the direction “along the outer lane line” as axis one, the direction “perpendicular to the outer lane line” as axis two, and the intersection of the “outer lane line” and the right edge of the image as the origin;

The mapping formula between camera coordinates and actual space coordinates is as follows:

Where L represents the distance from the camera focus to the actual measurement point; L_real represents the actual length of a point in space; L_pixel represents the pixel length of this point after taking a photo; f represents the focal length of the camera; cellSize represents the pixel size; f and cellSize are fixed values;

The calculation formula of L is as follows:

Among them, H0 represents the "vertical distance between the camera focus and axis 1", H1 represents the "axis 2 coordinate value of the measuring point", R0 represents the "vertical distance between the camera focus and axis 2", and R1 represents the "axis 1 coordinate value of the measuring point". Once the camera is fixedly installed, H0 and R0 become fixed values. See Figure 10.

Substituting formula 2 into formula 1, we get the new expression:

Ratio represents the conversion ratio from pixel coordinates to actual coordinates. As can be seen from formula 3:

Ratio＝f(H1,R1) (Formula 4)

Since H0, R0, f, and cell_size are fixed values, Ratio can be expressed as a function controlled by the H1 and R1 variables;

As can be seen from Figure 11, Formula 4 approximates a plane. In order to simplify the calculation, f(H1, R1) is converted into a plane function. The simplified specific formula is as follows:

Ratio＝a*H1+b*R1+c (Formula 5)

Among them, a/b/c are the parameters to be found;

S42, using the least squares method to fit, requires the introduction of multiple sets of known points

{(H1_0,R1_0,Ratio_0),(H1_1,R1_1,Ratio_1),......,

(H1_n, R1_n, Ratio_n)}, referred to as the fitting point set; each point is composed of a triplet, including the pixel coordinates of the point in the second direction of the axis, the pixel coordinates of the point in the first direction of the axis, and the scale of the point;

S43, determine the triplet, as shown in FIG14, through model detection and post-processing logic, the pixel coordinates of the axis of each vehicle target can be obtained;

According to formula 3, the actual wheelbase can be expressed as follows:

Among them, L _{real wheelbase} is the actual wheelbase of the vehicle; L _{pixel wheelbase} is the pixel wheelbase of the vehicle; d _x is the differential unit, as shown in Figure 13;

Approximate calculation of formula 7:

Since Formula 7 and Formula 8 are very close, the error can be ignored, so Formula 8 is subsequently decomposed to obtain the calculation formula of the triplet:

Among them, H1, R1, L _{pixel wheelbase} -----pixel-level coordinate data can be calculated based on model output + post-processing; L _{real wheelbase is} 2500mm;

S44, according to the triplet calculation method of step S43, based on the 200 sets of car tracking sequences for calibration reference, multiple triplet groups are calculated, thereby obtaining the fitting point set of step S42, as shown in FIG15;

The tracking sequences of 200 groups of cars for calibration reference are obtained as follows:

a. For each lane, 200 tracking sequences are selected as calibration references, as shown in Figure 12.

b. The tracking sequence is obtained by using target detection and target tracking.

c. Only the sedan category is used as a reference.

d. Use the car’s “pixel-level wheelbase” and “actual-level wheelbase” for calibration calculations.

e. The length range of car axles is "2400-2550mm", and this article intends to use 2500mm as a reference value for the actual value of the wheelbase of all cars.

S45, fitting,

a. The first fitting is based on all the triplets, fitting the plane, and calculating a set of (a, b, c); assuming that the parameters after calibration are (a, b, c) = (a1, b1, c1); substituting the above structure into formula 5. We get:

Ratio＝a1*H1+b1*R1+c1 (Formula 11)

b. Eliminate triplets with large deviations

Based on Formula 11 and Formula 9; the calculated deviation is:

Where pia represents the deviation of the scale calculated by formula 11 from the scale in formula 9;

Based on the deviations of all triples calculated by formula 12, the top 20% of triples with the largest deviations are deleted, and 80% of the triples are retained;

c. The second fitting, for the remaining triples in b, fit the plane and calculate a set (a, b, c); assume that the parameters after calibration are (a, b, c) = (a2, b2, c2). Substitute the above structure into formula 5; get:

Ratio＝a2*H1+b2*R1+c2 (Formula 12)

At this point, self-calibration is completed.

S5, determine the target vehicle, obtain its tracking sequence, axle information at different positions, and coordinate information, and based on the mapping relationship in step S4, obtain the estimated value of the final length of the target vehicle and the estimated value of the vehicle speed.

The calculation steps of the target vehicle's final length estimate are as follows:

S51, obtaining a tracking sequence of the target vehicle, and calculating the “pixel-level length” based on “key point 1, key point 2” of the vehicle at different positions in the sequence;

S52, based on formula 12, converting the length at the pixel level into the spatial length;

S53, averaging multiple spatial lengths in the sequence to obtain an estimated final length of the vehicle.

The calculation steps for the target vehicle speed calculation are as follows:

S54, obtaining a tracking sequence of the target vehicle, and calculating the projection coordinates of the center point of the front axle of the vehicle to the ground based on the axle center point results of the vehicle at different positions in the sequence;

S55, calculating the distance at the pixel level based on the “ground projection coordinates of the center point of the front axle” of the vehicle at the “first position” and the “ground projection coordinates of the center point of the front axle” of the vehicle at the “last position”;

S56, performing coordinate conversion based on formula 12 to obtain the actual distance between the “first position” and the “last position”;

S57, calculating the time difference between "the imaging time of the frame where the first position is located" and "the imaging time of the frame where the last position is located";

S58, based on the actual distance/imaging time difference between the front and rear ends, calculate the estimated value of the vehicle speed.

In step S5 of this embodiment, the vehicle width and the number of vehicle axles can also be calculated;

The steps to calculate the vehicle width are as follows:

Use "Key Point 2/Key Point 3" to calculate the vehicle pixel width;

Calculate the coordinates of the midpoint of the line connecting key point 2 and key point 3, and further calculate the intersection of the "vertical line from the center point to the left lane line" and the two lane lines; the pixel distance between the two intersection points represents the lane pixel width;

The vehicle width is calculated using the following formula: (vehicle pixel width/lane pixel width)*preconfigured lane line width;

The steps to calculate the number of vehicle axles are as follows:

Get the tracking sequence of the target vehicle based on the number of axles of the vehicle at different positions in the sequence;

Using the multi-frame voting method, if the number of occurrences of a certain "axle number" is greater than half of the total number of tracking sequences, and the "axle number" is greater than or equal to 2, it will be taken as the final result; otherwise, no axle detection result will be returned.

If only the "speed estimation/length estimation/axle number calculation" of the vehicle is needed, no manual labeling is required; if in addition to the "speed estimation/length estimation/axle number calculation", the "vehicle width" needs to be calculated, then the width value of each lane needs to be manually input. The present invention can automatically calibrate, has high accuracy, and reduces the complexity of manual operation.

Claims (9)

1. A monocular side angle imaging vehicle scale estimation and speed estimation method comprises the following specific steps:

S1, a target detection model is established, and a vehicle target, wheels, key points and lane lines are detected;

s2, target tracking is carried out, and a tracking sequence of each vehicle, a vehicle detection frame and a vehicle classification result are obtained;

S3, calculating coordinates of a wheel center point projected onto a road surface to obtain axle information, and dividing lane lines to obtain a plurality of lanes;

s4, based on the tracking sequence of the similar targets in the step S2 and the axle information of the similar targets in the step S3, performing self-calibration on the corresponding targets to acquire the mapping relation between the pixel coordinate systems and the actual coordinate systems of the similar targets; the specific steps of the self-calibration are as follows:

s41, setting a coordinate system, wherein each lane is defined by two lane lines, and takes the direction along the outer lane line as an axis I; taking the direction vertical to the outer lane line as a second axis; taking the intersection point of the outer lane line and the right boundary of the image as an origin;

the mapping formula of the camera coordinates and the actual space coordinates is as follows:

Wherein L represents "distance of camera focus to actual measurement point"; l _real represents "actual length under a certain point space coordinate"; l_pixel represents "pixel length after this point photographing"; f represents "camera focal length"; cellSize represents the "pixel size"; f and cellSize are fixed values;

The calculation formula of L is as follows:

Wherein H0 represents the vertical distance between the camera focus and the first axis, H1 represents the coordinate value of the second axis of the measuring point, R0 represents the vertical distance between the camera focus and the second axis, and R1 represents the coordinate value of the first axis of the measuring point; once the camera is fixedly installed, H0 and R0 become fixed values;

Bringing equation 2 into equation 1 yields a new expression:

Expressed in Ratio, the conversion Ratio of pixel coordinates to actual coordinates can be seen from equation 3:

Ratio=f (H1, R1) (formula 4)

Since H0, R0, f, cellSize are fixed values, ratio can be expressed as a function controlled by the H1 and R1 variables;

to simplify the calculation, f (H1, R1) is converted into a planar function, and the specific formula after simplification is as follows:

Ratio=a h1+b r1+c (formula 5), wherein a/b/c is the parameter to be solved;

s42, using least squares fitting, it is necessary to introduce multiple sets of known point sets { (h1_0, r1_0, ratio_0), (h1_1, r1_1, ratio_1),.

(H1_n, R1_n, ratio_n) }, abbreviated as fitting point set; each point is formed by a triplet and comprises a point axis two-direction pixel coordinate, a point axis one-direction pixel coordinate and a point scale;

s43, determining a triplet, and obtaining pixel coordinates of each vehicle target axle center through model detection and post-processing logic;

from equation 3, the expression for the actual wheelbase can be found as follows:

Wherein L _{real Wheelbase} is the actual wheelbase of the vehicle; l _{pixel Wheelbase} is the pixel wheelbase of the vehicle; d _x is a differential unit;

approximation calculation was performed for equation 7:

since equation 7 and equation 8 are very close, the error is negligible, so equation 8 is subsequently decomposed, where the triplet calculation equation is derived:

Wherein, the coordinate data of H1, R1, L _{pixel Wheelbase} - - -at the level of equal pixels can be calculated based on model output+post-processing; l _{real Wheelbase} mm;

s44, calculating a plurality of triples based on the tracking sequence of the 200 groups of cars with calibration reference according to the three-tuple calculation method in the step S43, so as to obtain a fitting point set in the step S42;

S45, fitting is carried out,

A. fitting for the first time, and calculating a group (a, b, c) based on all triples and fitting planes; assuming that the calibrated parameter is (a, b, c) = (a 1, b1, c 1); bringing the above structure into formula 5; the method comprises the following steps:

Ratio=a1 h1+b1 R1+c1 (formula 11)

B. triads with larger rejection bias

Based on equation 11 and equation 9; the calculated deviation is:

Wherein pia represents the deviation of the scale calculated by formula 11 from the scale in formula 9;

all triad deviations calculated based on equation 12; deleting the triples of which the number is the first 20% of the maximum deviation; 80% of triples remain;

c. fitting the remaining triples in the step b for the second time, fitting planes, and calculating to obtain a group (a, b, c); assuming that the calibrated parameter is (a, b, c) = (a 2, b2, c 2); bringing the above structure into formula 5; the method comprises the following steps:

Ratio=a2 h1+b2 r1+c2 (formula 13) to this point, the self calibration is completed;

S5, determining the target vehicle, acquiring tracking sequences, axle information and coordinate information of the target vehicle at different positions, and obtaining a final length estimated value of the target vehicle and an estimated value of the vehicle speed based on the mapping relation in the step S4.

2. A monocular side angle imaging vehicle dimension estimation and speed estimation method according to claim 1, wherein: the target detection model in the step S1 is obtained by expanding a hybrid head model based on yolov target detectors, and comprises a main network, wherein four output branches are arranged behind the main network, and each output branch corresponds to an up-sampling pyramid structure.

3. A monocular side angle imaging vehicle dimension estimation and speed estimation method according to claim 2, wherein: each frame of the first output branch, the second output branch and the third output branch is predicted, and the fourth output branch is predicted according to a set interval.

4. A monocular side angle imaging vehicle dimension estimation and speed estimation method according to claim 2, wherein: the acquired data set is divided into a lane line data set and a vehicle data set, and is marked separately.

5. A monocular side angle imaging vehicle dimension estimation and speed estimation method according to claim 2, wherein: the tracking sequence in step S2 is obtained based on the iou tracking mode and the output result of the output branch one.

6. A monocular side angle imaging vehicle dimension estimation and speed estimation method according to claim 2, wherein: the specific step of calculating coordinates of the wheel center point "projected onto" the road surface in step S3 includes:

S31, acquiring a straight line A based on the connection line of the key point 1 and the key 2;

s32, calculating the vertical point coordinates from the wheel center point to the straight line A, namely, the projection point coordinates of the wheel center.

7. A monocular side angle imaging vehicle dimension estimation and speed estimation method according to claim 1, wherein: the calculation step of the final length estimation value of the target vehicle in step S5 is as follows:

S51, acquiring a tracking sequence of a target vehicle, and calculating the length of a pixel layer based on the key points 1 and 2 of the vehicles at different positions in the sequence;

S52, converting the length of the pixel layer surface into a space length based on the formula 13;

And S53, averaging a plurality of space lengths in the sequence to obtain a final length estimation value of the vehicle.

8. A monocular side angle imaging vehicle dimension estimation and speed estimation method according to claim 1, wherein: the calculation step of the target vehicle speed calculation in step S5 is as follows:

S54, acquiring a tracking sequence of a target vehicle, and calculating projection coordinates from the center point of the forefront axle of the vehicle to the ground based on axle center point results of the vehicle in different positions in the sequence;

s55, calculating the distance of the pixel layer based on the ground projection coordinates of the central point of the front axle of the vehicle at the 'head position' and the ground projection coordinates of the central point of the front axle of the vehicle at the 'tail position';

s56, performing coordinate transformation based on the formula 13 to obtain the actual distance between the 'head position' and the 'tail position';

s57, calculating the time difference between the imaging moment of the frame where the head position is located and the imaging moment of the frame where the tail position is located;

s58, calculating an estimated value of the vehicle speed based on the head-to-tail actual distance/imaging time difference.

9. A monocular side angle imaging vehicle dimension estimation and speed estimation method according to claim 1, wherein: in the step S5, the vehicle width and the vehicle axle number can be calculated;

The vehicle width is calculated as follows:

calculating the vehicle pixel width by using the key point 2/key point 3;

calculating the midpoint coordinates of the connecting line of the key point 2 and the key point 3, and further calculating to obtain the intersection point of the perpendicular line from the center point to the left lane line and the two lane lines; the pixel distance between the two intersection points represents the lane pixel width;

The vehicle width is calculated by the following formula: (vehicle pixel width/lane pixel width) preconfigured lane line width;

The calculation steps of the vehicle axle number are as follows:

Acquiring a tracking sequence of a target vehicle, and based on the number of axles of the vehicle in different positions in the sequence;

Using a multi-frame voting mode, wherein the occurrence number of a certain 'number of axles' is more than half of the total number of tracking sequences, and the 'number of axles' is more than or equal to 2, and the final result is obtained; otherwise, the axle detection result is not returned.