CN111260957A - Lane departure warning system - Google Patents

Abstract

The invention relates to the technical field of lane line detection, in particular to a lane departure alarm system; it includes a road image acquisition module, a road image processing module and a deviation alarm module; the road image acquisition module includes an infrared camera and a CCD camera for acquiring real-time The road image processing module detects the lane line on the obtained road image, and the departure alarm module combines the time when the car deviates from the lane and the deviation angle of the car, and uses the curvature information of the detected lane line to carry out the detection of the car. Departure alarm, the present invention solves the problem of difficult lane line detection in complex road scenes such as road shadows, blurring, and occlusion, and significantly reduces the problem of false alarms on curved roads, vehicle aisle pressure lines, and vehicle steering. Strong robustness, high real-time performance, and strong creativity.

Description

A lane departure warning system

technical field

The invention relates to the technical field of lane line detection, in particular to a lane departure warning system.

Background technique

At present, most lane departure warning systems first obtain road images through on-board CCD cameras and infrared cameras, and the system host detects lane lines through image processing algorithms, and then calculates whether the current vehicle is on the current road. , and finally carry out a lane departure alarm according to the calculation result.

The lane detection module of the current lane departure system has poor robustness to lane line detection under complex road conditions, and is only suitable for comparison of regular and interference-free road conditions. In the actual complex road scene, such as the intermittent shadow situation, the boundary area between the shadow and the road is the place where the gray level suddenly changes, the traditional edge detection will erroneously extract it and treat it as a lane line detection process, causing errors road departure warning. On the other hand, although the applied hough transform has a better effect on the detection of straight pixels, in practice, curved roads are more common. When the car is driving on a curved road, the calculation of the road deviation by the system host will be different from the actual situation. If the difference is large, the road deviation alarm cannot be accurately performed. It can be seen that the accuracy and robustness of the traditional technology will be significantly reduced when encountering road bends, road shadows, occlusions, etc., and it is not universal.

In addition, most lane departure warning models are based on the current coordinate information of the car to measure whether the car deviates from the lane. When the distance between the car and the lanes on both sides is less than a certain safety distance threshold, it is judged that the car will deviate from the lane. This method is simple to calculate, but it will cause false alarms when driving closer to the lane line, and when the car suddenly deviates greatly in direction, it cannot correctly and timely issue a deviation alarm.

SUMMARY OF THE INVENTION

Aiming at the deficiencies of the prior art, the present invention discloses a lane departure warning system, in order to adapt to the road detection and road departure warning functions in various complex roads such as shadow, occlusion, blur and other road scenarios, and improve the accuracy of road departure warning , timeliness.

The present invention is achieved through the following technical solutions:

A lane departure alarm system, comprising a system host, which is used for information processing, storage, command sending and execution; further comprising a road image acquisition module, a road image processing module and a deviation alarm module; the road image acquisition module includes an infrared A camera and a CCD camera are used to obtain real-time road conditions images; the road image processing module detects the lane lines on the obtained road images, and the deviation alarm module combines the time when the car deviates from the lane and the car deviation angle, and uses all the Detect the curvature information of the lane line, and carry out the vehicle departure alarm.

Further, the road image processing module includes preprocessing, edge extraction, sliding window grouping and lane line fitting.

Further, the main processing of the preprocessing is to gray-scale the color road image, and the calculation formula is as follows:

Grey=R*0.299+G*0.587+B*0.114

where R, G, and B are red, green, and blue channels, respectively;

For image downsampling, the algorithm adopts the method of bilinear interpolation. The floating point coordinates of point (x, y) after scaling by bilinear interpolation are (x1+u, y1+v), where x1, y1 is the integer part of its floating-point coordinates, and u, v are the fractional parts of floating-point coordinates; the calculation formula is as follows:

Further, the edge extraction adopts a Gaussian smoothing filter, and the two-dimensional form is as follows:

where x, y are the distances from the center of the frequency rectangle, and σ is a measure of the spread about the center;

Use the sobel operator to extract the edge of the road image. The sobel operator includes two sets of matrix templates for horizontal and vertical convolution operations. The horizontal and vertical convolution templates are as follows:

The following two formulas calculate the gradient magnitude and gradient direction for each pixel, respectively:

Further, for the pixels obtained by edge extraction, the sliding window grouping uses the hog feature of the image to roughly locate the position of each lane line; then a rectangular area of a certain size is set, and the mean coordinates of the two starting points are used as the rectangle. The midpoint of the lower side of the area stores the coordinates of the pixel points in the rectangular area, takes the average value of the abscissas of the stored pixel points, and calculates the point coordinates as the coordinates of the midpoint of the lower side of the rectangle in the next window. Follow the above steps to continue searching until the All lane line pixels are grouped.

Further, the lane line fitting adopts the method of nonlinear regression of neural network to fit the road lane lines, sets a network hidden layer of 6 nodes, uses the tanh activation function to perform nonlinear change, and first performs linear transformation, the formula: as follows:

z=θ ₀ +θ ₁ x ₁ +θ ₂ x ₂ +…+θ ₆ x ₆

Among them, θ ₀ -θ ₆ is the weight, and the result is nonlinearly transformed by the tanh function. The formula is as follows:

The loss function adopts the mean square error MSE, and the formula is as follows:

where F(z _t ) is the actual value, and Pred _t is the predicted value;

The stochastic gradient descent method SGD is used to optimize the solution, and its mathematical formula is as follows:

where α is the learning rate, and _θi eventually converges to the maximum value.

Further, the deviation alarm module calculates the curvature of the lane in front of the car according to the detection result of the lane line. Due to the parallel characteristics of the two sides of the lane line, only one side needs to be calculated. _The calculation formula of the lane curvature K is as follows:

Among them, y is the function of the fitted lane line, and the curvature K _c of the driving trajectory of the car in the time t is calculated; K _p is the error value. Safe state, if the driver does not turn on the turn signal, and

|K _c -K _l |＞K _p

Then further calculate the time T from the current position of the car to the car leaving the current lane, and compare T with the set safety threshold time T1; T>T1, the car is currently in a safe state, T<T1, then the car will occur lane departure , the system host controls the Led alarm and the sound alarm to send an alarm signal to the driver.

Further, the time T is calculated by the following formula:

Among them, L _x is the lateral distance from the current car position to the lane line in the direction of the car’s deviation, V is the current driving speed of the car, and θ is the angle at which the current car deviates from the center longitudinal axis;

The formula for calculating _Lx is as follows:

Among them, L ₀ is the lateral distance from the current car center position to the lane line in the direction that the car deviates, W _l is the width of the lane, and W _c is the width of the car.

The beneficial effects of the present invention are:

The lane departure alarm model designed by the invention combines the time when the vehicle deviates from the lane and the vehicle deviation angle, and fully utilizes the curvature information of the detected lane line. It can effectively solve the problem of false alarms on curved roads, car aisle pressure lines and car steering, while greatly improving the accuracy of deviation alarms, solving the problem of difficult lane line detection in complex road scenes such as road shadows, blurring, and occlusion, and significantly reducing curvy roads. , car aisle pressure line and car steering error alarm problems. Strong robustness and high real-time performance.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

FIG. 1 is a block diagram of the overall flow of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1

Referring to FIG. 1 , the present embodiment discloses a lane departure warning system, including a system host, which is used for information processing, storage, command sending and execution; and a road image acquisition module, a road image processing module, and a departure warning module The road image acquisition module includes an infrared camera and a CCD camera to acquire real-time road conditions images; the road image processing module performs lane line detection on the acquired road images, and the deviation alarm module combines the detection of the vehicle deviating from the lane. Time and vehicle offset angle, using the curvature information of the detected lane line to carry out the vehicle departure alarm; the road image processing module includes preprocessing, edge extraction, sliding window grouping and lane line fitting

The purpose of image preprocessing is to save computation and obtain better algorithm processing effect. The main processing is to gray-scale the color road image and the calculation formula is as follows:

Grey=R*0.299+G*0.587+B*0.114

where R, G, and B are red, green, and blue channels, respectively;

For image downsampling, the algorithm adopts the bilinear interpolation method, this method will not cause image distortion, the processing effect is good, and the calculation amount is not too large. The floating point coordinates of point (x, y) scaled by bilinear interpolation are (x1+u, y1+v), where x1, y1 are the integer parts of its floating point coordinates, and u, v are floating point coordinates the fractional part of . The calculation formula is as follows:

Before edge extraction is performed on the processed image, since the image collected on the traffic road will inevitably be affected by uncertain factors such as bad weather, dust and cloudy days, there will be some noise and stray gradients. A certain amount of image enhancement is performed so as to obtain a better processing effect in the future. In this embodiment, a Gaussian smoothing filter is used, and its template coefficient decreases as the distance from the template center increases, which helps to suppress noise. without overly blurring the image. The two-dimensional form of the Gaussian filter is as follows:

where x, y are the distances from the center of the frequency rectangle, and σ is a measure of the spread about the center;

The rotational symmetry of the Gaussian filter and the single-lobe Fourier transform spectrum make the image edges containing both low-frequency components and high-frequency components not polluted by unwanted high-frequency signals, while retaining most of the desired signals , which is beneficial to the subsequent image edge extraction.

Then we use the sobel operator to extract the edge of the road image. The sobel operator contains two sets of matrix templates for horizontal and vertical convolution operations. The horizontal and vertical convolution templates are as follows:

The following two formulas calculate the gradient magnitude and gradient direction for each pixel, respectively:

For the pixels obtained by edge extraction, it needs to be further grouped to distinguish each lane line. This embodiment adopts the sliding window method. First, the positions of each lane line are roughly located by using the hog feature of the image. Then set a rectangular area of a certain size, use the average coordinates of the two starting points as the lower midpoint of the rectangular area, store the coordinates of the pixel points in the rectangular area, take the average of the stored abscissas of the pixel points, and calculate the point coordinates as the lower The coordinates of the midpoint below the rectangle of a window, continue to search according to the above steps, until all the lane line pixels are grouped.

Most of the interpolation methods for fitting curves are often not accurate enough to fit the curves. In this embodiment, a method of nonlinear regression performed by a neural network is used to obtain the fitted road lane lines. The hidden layer of the network with 6 nodes is set, and the tanh activation function is used for nonlinear change. First perform a linear transformation, the formula is as follows:

z=θ ₀ +θ ₁ x ₁ +θ ₂ x ₂ +…+θ ₆ x ₆

The result is nonlinearly transformed by the tanh function, and the formula is as follows:

The loss function adopts the mean square error MSE, and the formula is as follows:

The stochastic gradient descent method SGD is used to optimize the solution, and its mathematical formula is as follows:

The lane departure warning model designed in this embodiment combines the time when the car deviates from the lane and the deviation angle of the car, and makes full use of the curvature information of the detected lane line. It can effectively solve the problem of false alarms on curved roads, car aisle pressure lines and car steering, and at the same time greatly improve the accuracy of deviation alarms.

According to the detection result of the lane line, the curvature of the lane in front of the car is calculated. Due to the parallel characteristics of the two sides of the lane line, only one side needs to be calculated. The calculation formula of the lane curvature K _l is as follows:

Among them, y is the function of the fitted lane line, and the curvature K _c of the driving trajectory of the car in the time t is calculated; K _p is the error value. Safe state, if the driver does not turn on the turn signal, and

|K _c -K _l |＞K _p

Then further calculate the time T from the current position of the car to the car leaving the current lane, and compare T with the set safety threshold time T1; T>T1, the car is currently in a safe state, T<T1, then the car will occur lane departure , the system host controls the Led alarm and the sound alarm to send an alarm signal to the driver.

Further, the time T is calculated by the following formula:

Among them, L _x is the lateral distance from the current car position to the lane line in the direction of the car’s deviation, V is the current driving speed of the car, and θ is the angle at which the current car deviates from the center longitudinal axis;

The formula for calculating _Lx is as follows:

Among them, L ₀ is the lateral distance from the current car center position to the lane line in the direction that the car deviates, W _l is the width of the lane, and W _c is the width of the car.

Example 2

In this embodiment, first install the system host, infrared camera, CCD camera, sound alarm, and LED alarm on the car, configure the dependent libraries opencv, numpy, tensorflow, python, and matplotlib required by the system program, and start system host. The following is the specific implementation process of the lane departure warning system.

1. Process road images:

The camera captures road image data in real time, the original size is 2880*1500, and transmits it to the system host. The system host performs certain image processing to facilitate subsequent functional processing.

Use the cv2.VideoCapture() method to obtain a frame of road image, perform grayscale transformation on the color road image using the formula Gray=R*0.299+G*0.587+B*0.114, and perform corresponding downsampling operations on the image.

According to the road image structure taken by the actual vehicle, determine the area selected by the ROI, and determine the coordinates of the four rectangular points (x1, y1), (x2, y2), (x3, y3), (x4, y4). Calculates the matrix that transforms the rectangles in the source image to the rectangles in the destination image using a mathematical formula. Then use cv2.warpPerspective() to realize the perspective transformation of the image.

2. Road lane line extraction

The Sobel operator is used to extract the road edge of the image, which is used to extract the pixels of the road lane line. Before this, convert the image matrix to uint8 type for easy operation, and then use the cv2.Sobel() function to pass in the image data for edge extraction.

The next step is to group the extracted pixel points with lane lines. According to the hog feature of the extracted image, use the np.argmax() method to find the maximum value and determine the coordinates. Then set a rectangular window of the corresponding size, and use the np.mean() method to average the pixel coordinates of the lane lines contained in the rectangular window, thereby calculating the lower midpoint of the next rectangular window. Keep repeating the appeal steps until all searches are complete.

For lane line fitting, use the tf.Variable() function to declare the tensor of [1,6], use the tf.nn.tanh() function to perform nonlinear transformation, and use the tf.reduce_mean() function to build the loss function , and finally use tf.train.GradientDescentOptimizer() to train the parameters, get the optimal value and sess.run() to start the calculation graph.

3. Road departure alarm

The system host first obtains the departure angle, curvature, speed, and lane width of the current car, and then calculates whether the lane departure occurs according to the established lane departure model, and then controls the LED alarm and sound alarm according to the judgment result.

The lane departure warning model designed in this embodiment combines the time when the car deviates from the lane and the deviation angle of the car, and makes full use of the curvature information of the detected lane line. It can effectively solve the problem of false alarms on curved roads, car aisle pressure lines and car steering, while greatly improving the accuracy of deviation alarms, solving the problem of difficult lane line detection in complex road scenes such as road shadows, blurring, and occlusion, and significantly reducing curvy roads. , car aisle pressure line and car steering error alarm problems. Strong robustness and high real-time performance.

The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The recorded technical solutions are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (8)

1. A lane departure warning system comprises a system host, wherein the system host is used for information processing storage and command sending and execution; the system is characterized by comprising a road image acquisition module, a road image processing module and a deviation alarm module; the road image acquisition module comprises an infrared camera and a CCD camera and is used for acquiring real-time road condition images; the road image processing module detects lane lines of the acquired road image, and the departure warning module performs car departure warning by combining the time when the car departs from the lane and the car departure angle and utilizing the curvature information of the detected lane lines.

2. The lane departure warning system of claim 1, wherein the road image processing module includes preprocessing, edge extraction, sliding window grouping, and lane line fitting.

3. The lane departure warning system according to claim 2, wherein the preprocessing mainly processes a color road image by graying out a sum, and a calculation formula is as follows:

Grey＝R*0.299+G*0.587+B*0.114

r, G, B are respectively red, green and blue channels;

for the down sampling of the image, the algorithm adopts a bilinear interpolation method, the floating point coordinates of the point (x, y) which is scaled by the bilinear interpolation are (x1+ u, y1+ v), wherein x1 and y1 are integer parts of the floating point coordinates, and u and v are decimal parts of the floating point coordinates; the calculation formula is as follows:

4. the lane departure warning system according to claim 2, wherein the edge extraction employs a gaussian smoothing filter in two dimensions of:

where x, y are the distances from the center of the frequency rectangle, and σ is a measure of the spread about the center;

the method comprises the following steps of utilizing a sobel operator to carry out edge extraction work of a road image, wherein the sobel operator comprises two groups of matrix templates and is used for transverse and longitudinal convolution operation, and the transverse and longitudinal convolution templates are as follows:

the gradient magnitude and gradient direction of each pixel are calculated by the following two formulas:

5. the lane departure warning system according to claim 2, wherein the sliding window grouping roughly locates the position of each lane line using the hog feature of the image for the pixel points obtained by the edge extraction; and setting a rectangular area with a certain size, using the mean coordinates of the two starting points as the lower middle point of the rectangular area, storing the coordinates of the pixel points in the rectangular area, averaging the horizontal coordinates of the stored pixel points, calculating the coordinates of the points as the coordinates of the lower middle point of the rectangle of the next window, and continuously searching according to the steps until all the lane line pixel points are completely grouped.

6. The lane departure warning system according to claim 2, wherein the lane line fitting adopts a neural network to perform nonlinear regression to obtain a fitted lane line, a network hidden layer with 6 nodes is provided, a tanh activation function is used to perform nonlinear change, first linear transformation is performed, and the formula is as follows:

z＝θ₀+θ₁x₁+θ₂x₂+…+θ₆x₆

wherein theta is₀-θ₆The weight is taken as the weight, and the result is nonlinearly transformed by using a tanh function, and the formula is as follows:

the loss function adopts mean square error MSE, and the formula is as follows:

wherein F (z)_t) As actual value, Pred_tIs a predicted value;

the method adopts a random gradient descent method SGD to carry out optimization solution, and the mathematical formula is as follows:

where is the α learning rate, θ_iEventually converging to a maximum value.

7. The lane departure warning system according to claim 2, wherein the departure warning module calculates the curvature of the lane ahead of the vehicle based on the detection result of the lane line, only one side of the lane is calculated due to the parallel characteristic of the two sides of the lane line, and the curvature K of the lane_lThe calculation formula is as follows:

wherein y is a fitted lane line function, and the curvature K of the driving track of the automobile in the time t is calculated_c；K_pAs an error value, if the difference between the curvature of the lane and the curvature of the driving track of the automobile does not exceed the error value, the driver looks at the safety state of the driving position of the automobile, and if the driver does not turn on the turn signal lamp, the driver turns on the turn signal lamp

|K_c-K_l|＞K_p

Further calculating the time T from the current position to the time T when the automobile leaves the current lane, and comparing the T with the set safe threshold time T1; t is greater than T1, the automobile is in a safe state at present, T < T1, the automobile deviates from a lane, and the host computer of the system controls the Led alarm and the voice alarm to send out alarm signals to a driver.

8. The lane departure warning system according to claim 7, wherein the time T is calculated by the following equation:

wherein L is_xThe transverse distance from the current automobile position to the lane line of the automobile in the deviation direction is shown, V is the current running speed of the automobile, and theta is the angle of the current automobile deviating from the central longitudinal axis;

L_xthe calculation formula of (a) is as follows:

wherein L is₀The transverse distance, W, from the current center position of the vehicle to the lane line of the vehicle in the direction of departure_lIs the width of the lane, W_cThe width of the car.

Images