CN107229908B - A kind of method for detecting lane lines - Google Patents

Abstract

The present invention provides a kind of method for detecting lane lines, obtains image using the forward sight camera being installed on vehicle, handles in real time dynamic image, achieve the purpose that accurately identify detection to lane line.Method for detecting lane lines of the invention has the advantage that first is that applicability is more preferable, can identify the lane line in night, rainy day and tunnel；Second is that algorithm is simple and time-consuming shorter, it is able to satisfy the requirement of real-time；Third is that detection error is small, precision is high.

Description

A lane line detection method

Technical field: the present invention relates to smart car technology, in particular to a method for detecting car lane lines.

technical background:

With the progress of society, the improvement of people's living standards and the development of transportation tools, the number of cars in the world has increased dramatically. At the same time, traffic safety issues have also become a hot issue of global concern. Advanced Driver Assistance Systems (ADAS) have emerged and have been widely used. focus on. ADAS uses various sensors to obtain information inside and outside the vehicle, and after various processing, it finally alerts the driver to possible dangers that may occur in order to reduce the incidence of traffic accidents.

Due to the limited level of image recognition processing in the actual use of existing ADAS, problems often occur in the detection of lane lines, and it is difficult to have long-term long-distance stable operation performance, which limits the possibility of application of this type of system in specific environments sex.

Invention content:

In view of the above problems, the present invention provides a lane line detection method, which processes dynamic images in real time to achieve the purpose of accurately identifying and detecting lane lines.

A lane line detection method, comprising:

1. Obtain road images: capture road images through the vehicle camera;

2. Preprocess the image:

Including color image grayscale, intercepting effective information area, filtering and denoising, image grayscale enhancement, edge detection and repairing lane lines. The processing step of repairing lane lines is to first find the required "short line segment" For processing, if it has 4 consecutive white points in a certain direction, it is determined that this is a "short line segment", find all "short line segments", and record its direction; the second step, for each short line segment in If there is another short line segment similar to its direction within 6 pixels, we think it should be a straight line, and assign all pixels between the two short line segments to 1, that is, to The black dots in between become white dots, so as to achieve the purpose of connecting lines; all short line segments can be processed to repair the intermittent lane lines;

3. Identify and track lane lines:

1. Recognition of left and right lane lines:

First, reduce the number of intersection points required to detect straight lines and set the shortest value for detecting straight line segments;

Secondly, when the vehicle is driving in the lane, the left and right lane lines in the captured road image are usually distributed on both sides of the image, and the slope of the two lanes has a range. By analyzing and calculating a large number of road image samples, the polar angle of the extreme point and the slope are constrained in sequence. In practice, the value of the constraint is changed according to the installation position of the camera; the polar angle range of the constraint is -a ₁ °<θ<a ₂ °, that is, to find a straight line within this range, reduce the processing data, and the left lane The constraint range of the slope of the line is b ₁ <k ₁ <b ₂ , the constraint range of the slope of the right lane line is b ₃ <k ₂ <b ₄ , and the straight lines not within this range are deleted from the array;

Again, sort the detected line segments from long to short, pick out the first 10 line segments, pick out all the line segments if the total number of segments is less than 10, and then divide these line segments into three categories, the first category is the slope greater than zero and the slope is similar Straight line, the second type is a straight line whose slope is less than zero and the slope is similar, and the third type is all other straight lines. After that, according to the two principles of the longer the length, the greater the weight, and the lower the position, the greater the weight of the first type. All straight line segments are fitted to the right lane line, and all straight line segments in the second category are fitted to the left lane line, while the line segments in the third category are not processed;

2. Dynamic lane detection:

In two adjacent frames of images, due to the short distance traveled by the vehicle, the position of the two lane lines will not deviate too much; in the actual algorithm, the slope, intercept and two lane line detected in the previous frame Save the coordinates of the intersection point, then in the next frame, the angle of the lane line should be within 3° of the angle of the previous frame, and the difference between the intercept position and the data of the previous frame should be within 20 pixels. According to this, we can Knowing the approximate position of the lane line in the next frame, looking for it can reduce a lot of processing. In addition, the distance between the intersection points of the two lane lines in two adjacent frames should be within 15 pixels. Through this condition, the real lane markings;

4. Correct lane lines

To convert the lane line perspective image into a top view image, the general transformation formula of inverse perspective transformation is

Among them, [xyz] is the coordinates of each point of the original captured image, and [x'y'z'] is the coordinates of the corresponding points of the image after inverse perspective transformation, is the perspective transformation matrix, the transformation matrix middle

And [a ₃₁ a ₃₂ ] produce rigid body transformation, including translation, rotation, zoom in and out, produce a perspective effect,

Expanding the transformation formula, we get

[x'y'z']＝[a ₁₁ x+a ₁₂ y+a ₁₃ za ₂₁ x+a ₂₂ y+a ₂₃ za ₃₁ x+a ₃₂ y+a ₃₃ z] (4-2)

Rewriting the transformation formula gives

In the formula (4-3), since the image is a two-dimensional plane, z=1, let a ₃₃ =1, and there are 8 unknowns at this time, then only need to know the coordinates of 4 points before transformation and the coordinates of points after transformation The 4 point coordinates of , a total of 8 coordinates can be used to obtain the transformation matrix After the transformation matrix is obtained, each pixel in the original image is calculated and then corresponding to the new position;

In actual processing, only the starting point and the ending point of each lane line are changed, a total of four points, and then the starting point and the ending point are connected to obtain the lane line after inverse perspective transformation;

5. Lane Offset Calculation

1. Angle offset calculation

Suppose the offset angle of the left lane line is θ ₁ , the offset angle of the right lane line is θ ₂ , the offset angle of the center line of the lane line is θ ₀ , the angle between the center line of the two lane lines and the lane line is α, the two lanes The angle between the centerline of the line and the positive direction of the x-axis is β,

Eliminating the angle α, the calculation formula of the vehicle body offset angle θ can be obtained as

In formula (5-1), the larger the θ angle is, the larger the vehicle body offset is,

In the image, the center line of the lane shifts to the right. In practice, the vehicle shifts to the left of the lane. Therefore, the direction of the deviation angle of the body is related to the β angle. When 0°<β<90°, the body shifts to the left θ angle, when 90°<β<180°, the body is shifted to the right by θ angle;

2. Distance offset calculation

Assuming that the actual distance between two lane lines is x, and the length of the lane line in the shooting part is y, in the image, the distance between the two lane lines occupies u pixels, and the length of the lane line occupies v pixels, then the image There is a certain proportional relationship between u and v in the coordinate system and x and y in the overlooking coordinate system. Assuming that the ratio in the x-axis direction is λ, and the ratio in the y-axis direction is μ, there is a formula

Assume that the lower limit of the lane line is offset by d ₁ pixel from the center line, and the upper limit of the lane line is offset by d ₂ pixels from the center line; assuming the position of the front of the vehicle in the actual coordinate system, that is, the distance of the camera position offset from the center line of the lane is D, the distance between the front of the car and the lower limit position is L ₀ ,

According to the offset d ₁ of the lower limit and the angle offset θ, the relationship diagram in the image coordinate system is made, and it is obtained according to the trigonometric function formula

According to formula (5-2), we can know

When L>L ₀ , according to the similar triangle relationship, the offset distance is

D＝μd ₁ -L ₀ tanθ (5-5)

When L<L ₀ , according to the similar triangle relationship, the offset distance is

D＝μd ₁ -L ₀ tanθ (5-6)

In formulas (5-5) and (5-6), if the calculation result is a positive value, it means that the vehicle has shifted a distance to the left, and a negative value means that the vehicle has shifted a distance to the right.

The lane line detection method of the present invention has the following advantages: one is better applicability, and can identify lane lines at night, in rainy days and in tunnels; The error is small and the precision is high.

Description of drawings:

Accompanying drawing 1 is the schematic flow chart of the present invention.

Attached Figure 2a is the image before lane line repair.

Attached Figure 2b is the repaired image of lane lines.

Attached Figure 3a is the original image.

Accompanying drawing 3b is a top view obtained by inverse perspective transformation of Fig. 3a.

Accompanying drawing 4a is a schematic diagram of a vehicle on an actual road.

Accompanying drawing 4b is a reverse perspective transformation of the lane line in Fig. 4a into a top view.

Fig. 5 is a schematic diagram of lane line deviation.

Accompanying drawing 6a is the relationship diagram 1 in the image coordinate system of the offset d1 and the angle offset θ of the lower limit of the lane line.

Accompanying drawing 6b is the relationship diagram 2 in the image coordinate system of the offset d1 and the angle offset θ of the lower limit of the lane line.

Accompanying drawing 6c is the relationship diagram 3 in the image coordinate system of the offset d1 and the angle offset θ of the lower limit of the lane line.

Detailed ways:

The implementation method and principle of the present invention will be further described below in conjunction with examples and accompanying drawings.

A lane line detection method, comprising:

1. Obtain road images: capture road images through the vehicle camera;

2. Preprocess the image:

Color image grayscale, intercept effective information area, filtering and denoising, image grayscale enhancement, edge detection and repair of lane lines. After edge detection, the edges of lane lines are intermittent and not continuous, and need to be repaired. The processing step is to first find the "short line segment" we need, and process each white point. If there are 4 consecutive white points in a certain direction, it is judged as a "short line segment", and all the "short line segments" are found. segment” and record its direction. The second step is to search for each short line segment in its direction. If there is another short line segment similar to its direction within 6 pixels, we think it should be a straight line. The pixel points of are assigned a value of 1, that is, the black dots between them are changed to white dots, so as to achieve the purpose of connection. By processing all short line segments, intermittent lane lines can be repaired.

Figure 2a is the edge detected by the Prewitt operator on the road image. It can be seen from the figure that the lane lines after edge detection are intermittent and not continuous, which may have a certain impact on the subsequent recognition of lane lines, so we Lane lines need to be repaired. The basic idea of repairing the lane line is to judge the distance between the two line segments. When the distance is less than the value we specified, assign the point between the two line segments a value of 1, that is, change the black point between them into a white point. , so as to achieve the purpose of connecting. The repaired image is shown in Figure 2b. It can be seen that the original discontinuous straight line segments are connected into a complete straight line.

3. Identify and track lane lines:

1. Recognition of left and right lane lines:

First of all, reduce the number of intersections required to detect straight lines and set the shortest value of detected straight line segments, which can increase the error tolerance rate of detected straight lines and allow some straight lines to be detected, which is conducive to improving the detection of dashed lanes. success rate of the line.

Secondly, when the vehicle is driving in the lane, the left and right lane lines in the captured road image are usually distributed on both sides of the image, and the slope of the two lanes has a range. By analyzing and calculating a large number of road image samples, the polar angle and slope of the extreme points are constrained in turn. In practice, the constrained value changes according to the installation position of the camera. In this example, the constrained polar angle range is -70°<θ<70°, that is, to find a straight line within this range and reduce the processing data. The constraint range for the slope of the left lane line is 0.355<k ₁ <0.73, and the constraint range for the slope of the right lane line is k ₂ <0.4516. Straight lines not within this range are deleted from the array, which can greatly reduce noise line segments.

Again, sort the detected line segments from long to short, pick out the first 10 line segments, and pick out all line segments if the total number of segments is less than 10. Then divide these line segments into three categories, the first category is straight lines with slopes greater than zero and similar slopes, the second category is straight lines with slopes less than zero and similar slopes, and the third category is all other straight lines. Afterwards, according to the two principles of the longer the length, the greater the weight, and the lower the position, the greater the weight, we fit all the straight line segments in the first category to the right lane line, and fit all the straight line segments in the second category to The left lane line, and the line segments in the third category are not processed because these are mostly noisy line segments.

2. Dynamic lane detection

In actual complex road conditions, it is not always possible to detect the left and right lane lines, nor is the detected lane line accurate. We should add some constraints.

In our country, the video taken by the camera is generally 25 frames per second, using the PAL system, that is, 25 pictures are taken per second. In ADAS with very high real-time requirements, it is generally processed frame by frame. Then the interval between each two pictures is 0.04 seconds, even if the vehicle is assumed to travel at a speed of 120Km/h, the vehicle can only move forward 1.33 meters. Therefore, in two adjacent frames of images, due to the short distance traveled by the vehicle, the positions of the two lane lines will not deviate too much. In the actual algorithm, we save the lane line slope, intercept, and intersection point coordinates of the two lane lines detected in the previous frame. Then in the next frame, the angle of the lane line should be within 3° of the angle of the previous frame. The difference between the intercept position and the data of the previous frame should be within 20 pixels. Based on this, we can know the approximate position of the lane line in the next frame, and finding it can reduce a lot of processing. In addition, the distance between the intersection points of two lane lines in two adjacent frames should be within 15 pixels. Through this condition, we can filter out the real lane lines. Therefore, motion detection can have higher accuracy using less processing time.

4. Correct lane lines

After the camera tilts to shoot an object, the formed image will be deformed, and the camera projects the actual three-dimensional scene onto the two-dimensional plane of the image. This projection is called perspective transformation. In actual situations, the camera is not necessarily installed in the middle, and the camera may be shifted left and right for some reasons. The distortion of lane lines is not conducive to our subsequent calculation of lane offsets, so we convert the perspective image into an overhead image. This process is called inverse perspective transformation.

To convert the lane line perspective image into a top view image, the general transformation formula of inverse perspective transformation is

Among them, [xyz] is the coordinates of each point of the original captured image, and [x'y'z'] is the coordinates of the corresponding points of the image after inverse perspective transformation, is the perspective transformation matrix. transformation matrix middle And [a ₃₁ a ₃₂ ] produce rigid body transformation, including translation, rotation, zoom in and out, produce a perspective effect,

Expanding the transformation formula, we get

[x'y'z']＝[a ₁₁ x+a ₁₂ y+a ₁₃ za ₂₁ x+a ₂₂ y+a ₂₃ za ₃₁ x+a ₃₂ y+a ₃₃ z] (4-2)

Rewriting the transformation formula gives

In the formula (4-3), since the image is a two-dimensional plane, z=1. Let a ₃₃ =1, and there are 8 unknown quantities at this time, then only need to know the coordinates of 4 points before transformation and 4 points after transformation, a total of 8 coordinates can obtain the transformation matrix After the transformation matrix is obtained, each pixel in the original image is calculated and then corresponding to the new position.

The coordinates of the four white points in Figure 3a are the coordinates of the original image, plus the coordinates of the four transformed vertices we want, the transformation matrix can be calculated by formula (4-3). After the transformation matrix is obtained, each pixel in the original image is calculated and then corresponding to the new position. In the new image, there may be some points that are not corresponding to the original image, that is, "holes", we can use the nearest neighbor interpolation method, or use bilinear interpolation or bicubic interpolation to fill the "holes" Complete. The top view obtained through inverse perspective transformation is shown in Figure 3b. It can be seen that the image after inverse perspective transformation eliminates the influence of deformation and restores the original appearance of the top view.

5. Lane Offset Calculation

1. Angle offset calculation

Assuming that the vehicle is currently on the road in Figure 4a, the angle at which the vehicle deviates from the lane line is θ. In an ideal situation, we transform the reverse perspective of the lane lines into a top view, as shown in Figure 4b, the two lane lines should be parallel, that is, the slope of the left lane line is equal to the slope of the right lane line. However, in actual situations, the bumps of the car and the slope of the road will make the two lane lines not necessarily parallel. At this time, we use the centerline of the two lane lines to calculate to reduce the error. As shown in Figure 5, suppose the offset angle of the left lane line is θ ₁ , the offset angle of the right lane line is θ ₂ , the offset angle of the center line of the lane line is θ ₀ , and the angle between the center line of the two lane lines and the lane line is α, the angle between the centerline of the two lanes and the positive direction of the x-axis is β, and

Eliminating the angle α, the calculation formula of the vehicle body offset angle θ can be obtained as

In formula (5-1), the larger the θ angle is, the larger the body offset will be.

And according to the small hole imaging principle of the camera, the centerline of the lane in the image shifts to the right, and in practice it appears that the vehicle shifts to the left of the lane, so the direction of the deviation angle of the vehicle body is related to the β angle. When 0°<β<90°, the body shifts to the left by the angle θ, and when 90°<β<180°, the body shifts to the right by the angle θ.

2. Distance offset calculation

After transformation, we obtained the top-view lane line image. Assume that the actual distance between the two lane lines is x, and the length of the lane line in the shooting part is y. In the image, the distance between the two lane lines occupies u pixels, and the lane The line length occupies v pixels. Then u and v in the image coordinate system have a certain proportional relationship with x and y in the overlooking coordinate system. Assume that the ratio in the x-axis direction is λ, and the ratio in the y-axis direction is μ. The formula

Assume that the lower limit of the lane line is offset by d ₁ pixel from the center line, and the upper limit of the lane line is offset by d ₂ pixels from the center line. Assume that in the actual coordinate system, the position of the head of the vehicle (that is, the position of the camera) is offset by a distance of D from the centerline of the lane, and the distance between the head of the vehicle and the lower limit position is L ₀ .

According to the offset d ₁ of the lower limit and the angle offset θ, the relationship diagram in the image coordinate system is made, as shown in Figure 6a, which is obtained according to the trigonometric function formula

According to formula (5-2), we can know

When L>L ₀ , as shown in Figure 6b, according to the similar triangle relationship, the offset distance is

D＝μd ₁ -L ₀ tanθ (5-5)

When L<L ₀ , as shown in Figure 6c, according to the similar triangle relationship, the offset distance is

D＝μd ₁ -L ₀ tanθ (5-6)

In formulas (5-5) and (5-6), if the calculation result is a positive value, it means that the vehicle has shifted a distance to the left, and a negative value means that the vehicle has shifted a distance to the right.

Claims (1)

1. A lane line detection method, characterized in that, 1. Obtain road images: capture road images through the vehicle camera; 2. Preprocess the image: Including color image grayscale, intercepting effective information area, filtering and denoising, image grayscale enhancement, edge detection and repairing lane lines. The processing step of repairing lane lines is to first find the required "short line segment" For processing, if it has 4 consecutive white points in a certain direction, it is determined that this is a "short line segment", find all "short line segments", and record its direction; the second step, for each short line segment in If there is another short line segment similar to its direction within 6 pixels, we think it should be a straight line, and assign all pixels between the two short line segments to 1, that is, to The black dots in between become white dots, so as to achieve the purpose of connecting lines; all short line segments can be processed to repair the intermittent lane lines; 3. Identify and track lane lines: 1. Recognition of left and right lane lines: First, reduce the number of intersection points required to detect straight lines and set the shortest value for detecting straight line segments; Secondly, when the vehicle is driving in the lane, the left and right lane lines in the captured road image are usually distributed on both sides of the image, and the slope of the two lanes has a range. By analyzing and calculating a large number of road image samples, the polar angle of the extreme point and the slope are constrained in turn. In practice, the value of the constraint is changed according to the installation position of the camera; the polar angle range of the constraint is -70°<θ<70°, that is, to find a straight line within this range, reduce the processing data, and adjust the slope of the left lane line The constraint range of is 0.355<k ₁ <0.73, the constraint range of the slope of the right lane line is k ₂ <0.4516, and the straight lines not within this range are deleted from the array; Again, sort the detected line segments from long to short, pick out the first 10 line segments, pick out all the line segments if the total number of segments is less than 10, and then divide these line segments into three categories, the first category is the slope greater than zero and the slope is similar Straight line, the second type is a straight line whose slope is less than zero and the slope is similar, and the third type is all other straight lines. After that, according to the two principles of the longer the length, the greater the weight, and the lower the position, the greater the weight of the first type. All straight line segments are fitted to the right lane line, and all straight line segments in the second category are fitted to the left lane line, while the line segments in the third category are not processed; 2. Dynamic lane detection: In two adjacent frames of images, due to the short distance traveled by the vehicle, the position of the two lane lines will not deviate too much; in the actual algorithm, the slope, intercept and two lane line detected in the previous frame Save the coordinates of the intersection point, then in the next frame, the angle of the lane line should be within 3° of the angle of the previous frame, and the difference between the intercept position and the data of the previous frame should be within 20 pixels. According to this, we can Knowing the approximate position of the lane line in the next frame, looking for it can reduce a lot of processing. In addition, the distance between the intersection points of the two lane lines in two adjacent frames should be within 15 pixels. Through this condition, the real lane markings; 4. Correct lane lines To convert the lane line perspective image into a top view image, the general transformation formula of inverse perspective transformation is Among them, [xyz] is the coordinates of each point of the original captured image, and [x'y'z'] is the coordinates of the corresponding points of the image after inverse perspective transformation, is the perspective transformation matrix, the transformation matrix middle And [a ₃₁ a ₃₂ ] produce rigid body transformation, including translation, rotation, zoom in and out, produce a perspective effect, Expanding the transformation formula, we get [x'y'z']＝[a ₁₁ x+a ₁₂ y+a ₁₃ za ₂₁ x+a ₂₂ y+a ₂₃ za ₃₁ x+a ₃₂ y+a ₃₃ z] (4-2) Rewriting the transformation formula gives In the formula (4-3), since the image is a two-dimensional plane, z=1, let a ₃₃ =1, and there are 8 unknowns at this time, then only need to know the coordinates of 4 points before transformation and the coordinates of points after transformation The 4 point coordinates of , a total of 8 coordinates can be used to obtain the transformation matrix After the transformation matrix is obtained, each pixel in the original image is calculated and then corresponding to the new position; In actual processing, only the starting point and the ending point of each lane line are changed, a total of four points, and then the starting point and the ending point are connected to obtain the lane line after inverse perspective transformation; 5. Lane Offset Calculation 1. Angle offset calculation Suppose the offset angle of the left lane line is θ ₁ , the offset angle of the right lane line is θ ₂ , the offset angle of the center line of the lane line is θ ₀ , the angle between the center line of the two lane lines and the lane line is α, the two lanes The angle between the centerline of the line and the positive direction of the x-axis is β, Eliminating the angle α, the calculation formula of the vehicle body offset angle θ can be obtained as In formula (5-1), the larger the θ angle is, the larger the vehicle body offset is, In the image, the center line of the lane shifts to the right. In practice, the vehicle shifts to the left of the lane. Therefore, the direction of the deviation angle of the body is related to the β angle. When 0°<β<90°, the body shifts to the left θ angle, when 90°<β<180°, the body is shifted to the right by θ angle; 2. Distance offset calculation Assuming that the actual distance between two lane lines is x, and the length of the lane line in the shooting part is y, in the image, the distance between the two lane lines occupies u pixels, and the length of the lane line occupies v pixels, then the image There is a certain proportional relationship between u and v in the coordinate system and x and y in the overlooking coordinate system. Assuming that the ratio in the x-axis direction is λ, and the ratio in the y-axis direction is μ, there is a formula Assume that the lower limit of the lane line is offset by d ₁ pixel from the center line, and the upper limit of the lane line is offset by d ₂ pixels from the center line; assuming the position of the front of the vehicle in the actual coordinate system, that is, the distance of the camera position offset from the center line of the lane is D, the distance between the front of the car and the lower limit position is L ₀ , According to the offset d ₁ of the lower limit and the angle offset θ, the relationship diagram in the image coordinate system is made, and it is obtained according to the trigonometric function formula According to formula (5-2), we can know When L>L ₀ , according to the similar triangle relationship, the offset distance is D＝μd ₁ -L ₀ tanθ (5-5) When L<L ₀ , according to the similar triangle relationship, the offset distance is D＝μd ₁ -L ₀ tanθ (5-6) In formulas (5-5) and (5-6), if the calculation result is a positive value, it means that the vehicle has shifted a distance to the left, and a negative value means that the vehicle has shifted a distance to the right.