JP7164342B2 - DRIVER ASSISTANCE SYSTEM AND GUARDRAIL DETECTION METHOD - Google Patents

Description

The present invention relates to a driver assistance system and a guardrail detection method.

Roadside guardrail detection is critical for highly automated driving. These structures (ie guardrails) are found along roadsides in various countries of the world. Their presence on the roadside physically limits the free space available for vehicles to move laterally. Therefore, detecting guardrails is very important.

This problem (ie guardrail detection) can be solved by using a camera (eg a monocular camera).

The present invention is a computer-implemented method for generating guardrail patterns:
a. capturing an image frame using a camera;
b. determining that an image frame includes at least two guardrail structures having a non-vertical portion NVP and a vertical portion VP of guardrails, and using a clipping box to track the guardrail structures within the image frame;
c. processing cutout box information to extract angular slope information of one of the tracked guardrail structures;
d. clustering the gradient information using a clustering method that produces clusters of information about the angles of the components of the clipping box;
e. extracting information about the non-vertical portion NVP of the clipping box;
f. repeating steps c-e until all of the tracked guardrail structures have gone through steps c-e;
g. and fitting a curve through the NVP of the tracked guardrail structure.

The angular slope information is an angular slope descriptor, preferably of at least two guardrail structures, providing slope lines of vertical and non-vertical portions of a single structure of the guardrail and information about the angular slope of the non-vertical portions with respect to the reference The histogram of oriented gradients may be extracted by providing HoG descriptors.

Angular gradient information may be extracted by implementing VODCA, a view-oriented, distributed, cluster-based approach to parallel computing.

The method has the advantage that the VODCA components for extracting HoG features can be reused to reduce the computational hardware load, and the classification components from the VODCA framework can be reused. ing.

Step g) may include interpolating a curve through the detected guardrail structure to obtain a continuous curve indicative of the guardrail.

The curve may be a polynomial curve through the intersections of sloped lines of NVPs of said at least two guardrail structures.

Step d. of the above method. may provide the mean μ at each non-vertical portion and the corresponding spread σ, which is used to optimize the curve fitting.
Step d. of the above method. may further include removing false positive guard structures.

It may be determined that the image frame contains two guardrail structures on different sides of the road. Thus, two curves may be generated.

The method therefore has the particular advantage of being able to detect multiple guardrails within a frame. Tracking of the detected guardrail structure is performed using any suitable tracking algorithm or method, in particular a Kalman filter.

To determine that an image frame contains two guardrail structures on different sides of the roadway, the angular information of the guardrail structures may be considered.
The method may further comprise warning the driver when the driver is approaching the guardrail.

Therefore, the method further provides that the disclosed method can assist lane departure systems and/or radar to (accurately) determine road boundaries, since guardrails are always along road boundaries and vice versa. It has several advantages.

The present invention also provides a driver assistance system for detecting guardrail structures:
a. a camera configured to capture a plurality of images;
b. a processor configured to be connected to a camera and configured to perform the operations described in the method above.

The driver assistance system may further comprise a radar, and the processor is configured to use information provided by the radar to generate the guardrail curve.

The invention also relates to a vehicle with a driver assistance system.

The vehicle may be a car, motorcycle, or truck.

The present invention also provides a computer-implemented machine learning method for determining that an image frame contains a guardrail structure:
a. providing a labeled dataset of at least one image frame of guardrail structures in a database and at least one machine being a learning model;
b. processing at least one image frame to extract angular gradient information of the guardrail structure of the guardrail;
c. and using angular gradient information to train said learning model for classifying image frames containing guardrail structures within the image frames.

The learning model may be any suitable machine learning approach, method or model, such as the support vector machine SVM or the adaptive boosting machine AdaBoost.

Computer-implemented machine learning method step b. may include implementing VODCA, a view-oriented, distributed, cluster-based approach to parallel computing.

The invention also relates to a computer system comprising a memory and a processor, wherein the processor is configured to perform the operations of the computer-implemented machine learning method described above.

FIG. 1 shows a repeating pattern within a guardrail. FIG. 2 shows a single construction of the guardrail. FIG. 3 shows the slope of a single construction of the guardrail. FIG. 4 shows the histogram of oriented gradient descriptor (HoG) feature representation of the guardrail structure. FIG. 5 is a flow diagram for training a machine learning algorithm for detecting and classifying guardrail structures. FIG. 6 shows the angle information contained in the clipping box. FIG. 7 shows the Gaussian mixture model density distribution of the angular information shown in FIG. FIG. 8 shows a polynomial fitting to the detected guardrail structure. FIG. 9 relates to a flow diagram of a two-step approach involving (1) detection of guardrail structures using a machine learning approach (S1-S5) and (2) generation of guardrail patterns from the detected guardrail structures. FIG. 10 shows the computer system of the present invention.

DETAILED DESCRIPTION Reference will now be made in detail to the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. Aspects of the disclosure are described below and the drawings are referred to to illustrate the teachings of the disclosure.

Before describing the present disclosure in detail, it is to be understood that the present invention is not limited in its application to the design details and arrangement of components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology used in this disclosure is for the purpose of description and should not be regarded as limiting. The indefinite articles "a" and "an" and the definite article "the" also refer to the corresponding plural. Headings are provided only to assist in understanding the invention and should not be construed as delimiting content between sections.

Detection of Guardrail Structures Guardrails 101 are linear structures on the roadside of highways and other types of roads 102 .

The guardrail structure is indicated by T1, T2, . . . It consists of a repeating pattern denoted as T6. Here it is important to understand that such repetitive patterns can be found using a camera (eg a monocular camera).

A guardrail repeating pattern, referred to herein as a guardrail structure (eg, T1-T6 in FIG. 1), may consist of two portions, a vertical portion VP and a non-vertical portion NVP shown in FIG.

A digital image of the guardrail structure may be processed to provide an angular gradient or similar feature descriptor of the image (as shown in FIG. 3). These descriptors may be used to detect such structures. For example, FIG. 4 shows the same guardrail structure as FIGS. 2 and 3, but visualized using the gradient orientation histogram HoG descriptor of the guardrail structure. Angular gradient descriptors (e.g., HoG, edge orientation histograms, scale-invariant feature transform descriptors, and shape context) may be used because the angular information provided by the angular gradient descriptors richly describes the object. is.

Trained Machine Learning Algorithms for Detecting Guardrail Structures The angular gradient descriptors provided above for guardrail structures can be used to provide trained machine learning algorithms for detecting guardrail structures. The resulting trained machine learning model can detect and classify guardrails and/or guardrail structures in images.

A method for training a machine learning algorithm for detecting guardrail structures is shown in FIG. The methods can be computer-implemented, and can be performed by any suitable computer system, such as a personal computer, laptop, mainframe, or any other system having at least a processor and memory. The training method and system may not be part of the driver assistance system included in the vehicle, thus training may be performed "off-line", ie when the method or system is not operating as a driver assistance method or system. "Offline" means real-time operation of the driver assistance method or system, ie the method or system is attempting to detect guardrail structures and/or guardrails while the vehicle is in operation.

The method may include the following steps. A data set is provided S501 to a database containing at least one digital image containing guardrails and/or guardrail structures, ie known guardrail structures. The database may contain a single digital image of the guardrail or a sequence of frames (eg, video) showing the guardrail structure. The dataset indicates the guardrail structures shown in the digital image and can be labeled to indicate different guardrail structures.

The dataset can be used to extract angular gradient features using the angular gradient descriptors S501. Angular gradient descriptors may be input into a recognition system based on supervised learning. For example, the recognition system may be a learning model or classifier, such as the support vector machine SVM or the adaptive boosting machine AdaBoost. The output of the method is a trained classifier S503 that can provide a list of guardrail structures present in the frame. The method may include a tracking step that tracks the detected structure and can provide a bounding box or clipping box around the guardrail structure.

Driver assistance method and system 1 . Guardrail Detection Method and System This trained model or classifier can be used to detect guardrail structures in any digital image online. When applying a trained model or classifier, at least two tracked guardrail structures can be detected. These structures can be verified as true-positive guardrail structures using a clustering algorithm, thus removing false-positively identified guardrail structures, if necessary. Additionally, a clustering algorithm can be used to determine the angular distribution of the classified guardrail structures.

In particular, the method or system may include removing false positive guardrail structures as described below. False positives may be removed by applying a clustering algorithm. The clustering algorithm may be a Gaussian mixture model (eg, one-dimensional Gaussian mixture model GMM-1D) or k-means.

This step is performed as follows. A sanity check can be applied to eliminate false positives. This may be achieved using the following method steps. A tracker can be used to track the detected structure. Guardrail structures (for further processing) may be considered. The track length may be greater than one. Track length relates to the number of instances of an object detected in a given track. In the present case, these objects are the potential guardrail structures of the guardrail, i.e. the objects have not yet undergone a sanity check. A bounding box containing the true positive guardrail structure may be obtained from a trained machine learning method classifier. Directional gradient information may then be extracted from the edge map for that region in a bounding box (similar to boxes T1-T6 in FIG. 1; FIG. 6 shows the contents of one bounding box).
Since the guardrail structure consists of NVP and VP components (see FIGS. 2 and 6), we may obtain the 0 degree or near 0 degree edges of the VP component in the directional gradient descriptor, eg, as shown in FIG. Here, the NVP edge component may contribute non-zero angular gradient information.

In this case, the directional gradient information values may be transferred to a clustering algorithm, as described above. Therefore, the two-dimensional angular information may be transformed into a row-major matrix or any suitable method to obtain one-dimensional data. From a clustering algorithm such as GMM, the distribution, ie the m Gaussian kernels with the highest probability, is selected for further analysis. One of the Gaussian distributions may correspond to the VP component and the other is related to the NVP component.

For example, m may be determined by setting a threshold T, and all Gaussian distributions exceeding the predetermined threshold T may be considered.

In the example cutout shown in FIG. 6, the GMM clustering output for n=4 (where n is the number of Gaussian kernels) is shown in FIG.

Table 1 shows the mean and spread of the guardrail structure shown in FIG. However, Table 1 only serves to illustrate and explain the invention. From FIG. 7 and corresponding Table 1, it can be observed that the gradient has a component (also called kernel) centered on the mean value along the spread indicated by sigma. It can be observed that the figure has two high density peaks: c3 at 2.46 degrees and c2 at -19.16 degrees.

Furthermore, since there is overlap between these components, they can be considered as one component. The other two components, c1 and c4, represent a pair of [-180, -180] circular representation lines. These two components belong to the guardrail structure and its angle can be determined.

Here, a sanity check may be applied to remove false positive guardrail structures/bounding boxes, eg by checking the GMM distribution. One of the kernel averages (out of all) may have to be 0°±Δ, where Δ is a predetermined threshold. Alternatively, the detected box can be removed.

The same operation may be performed for all detected guardrail structures (tracked with cutout boxes). However, any suitable tracking method may be used (eg Kalman filter) when implementing the method of the present invention. From these guardrail structures, we can extract the NVP angle information, ie the average μ above. These average values are relevant because the NVP component contributes to continuity between guardrail structures as follows.

2. Generating Guardrail Patterns from Detected Guardrail Structures To determine a guardrail pattern, a curve must be drawn through a set of points (i.e., for this purpose all of the guardrail structures contribute to one point). ). This point can be obtained as the intersection of the NVP gradient lines within the detected box.

Then, considering the angle of the NVP portion of the sensed structure, specifically the NVP, a directional (eg, polynomial) curve is fitted through the guardrail structure.

Optionally, a continuity check can be performed. Angular information (ie, μ) can be obtained using diffusion (ie, σ), as described above. This allows the stipple/drawn line to be adapted while considering information about the angle of NVP (within σ) of each guardrail structure. In this way, a smooth line 801 can be provided, which may be visualized in an image containing, for example, road 802 (see FIG. 8), or provided for other applications.

Drawing the curve while considering the angles of the NVP components ensures a robust implementation of the proposed algorithm in scenarios with guardrails on both sides of the road. In such situations, the driver assistance system or method can detect guardrail structures on both sides. In these scenarios, the disclosed method and system result in different NVP values for the guardrail structures found on the left and right sides of the road. As a result, the disclosed method and system allow two guardrails to be detected and/or drawn simultaneously within a given frame.

FIG. 9 shows an overview of the steps of the disclosed driver assistance system and method.

In step S901, a frame (ie, an image or video frame) is input. A trained machine learning algorithm or method is applied S902 to frames that detect guardrail structures. The guardrail structure of the guardrail is then tracked by any suitable tracking method (eg Kalman filter) and cutout boxes are provided (extracted) for further processing. From the information contained in the cutout box, the angular information of the structure in the cutout box is obtained S903, including the angular information about the NVP. Optionally, clustering can be applied to remove false positives S904. Information about the NVP angle is obtained and provided S905. Considering the obtained NVP angle information, a (eg, polynomial) curve is fitted through each guardrail structure.

FIG. 10 illustrates a computer system 1000 that implements the guardrail pattern generation method of the present invention. The computer system has a camera 1002 that captures frames that are processed by processor 1001 . Optionally, system 1000 has memory to store the captured and processed data.

The computer-implemented machine learning method of the present invention can be implemented in the same system, in which case camera 1002 is optional.

The computer system of the present invention may be a personal computer, an application specific integrated circuit ASIC, or a field programmable gate array FPGA.
Although this application relates to the invention described in the claims, it includes the following as another aspect.
1. A computer-implemented method for generating a guardrail pattern comprising:
a. capturing an image frame with a camera;
b. determining that the image frame includes at least two guardrail structures having a non-vertical portion NVP and a vertical portion VP of guardrails, and using a clipping box to track the guardrail structures within the image frame;
c. processing the cutout box information to extract angular slope information of one of the tracked guardrail structures;
d. clustering the angular gradient information using a clustering method that produces clusters of information about angles of the clipping box components;
e. extracting information about the non-vertical portion NVP of the clipping box;
f. repeating steps c-e until all of the tracked guardrail structures have gone through steps c-e;
g. and fitting a curve through said NVP of a tracked guardrail structure.
2. The angular gradient information is an angular gradient description providing gradient lines of the vertical and non-vertical portions of a single structure of guardrails of the at least two guardrail structures and information about the angular gradients of the non-vertical portions relative to a reference. A method according to claim 1, extracted by providing a child, preferably a gradient oriented histogram descriptor.
3. 3. The method of claim 1 or 2, wherein the angular gradient information is extracted by implementing VODCA, a view-oriented, distributed, cluster-based approach to parallel computing.
4. 4. The method of any one of claims 1-3, wherein step g) includes interpolating a curve through the detected guardrail structure to obtain a continuous curve indicative of the guardrail.
5. The method according to any one of the preceding paragraphs 1 to 4, wherein said curve is a polynomial curve through the intersections of sloped lines of NVPs of said at least two guardrail structures.
6. step d. provides the mean μ at each non-vertical portion and the corresponding spread σ, said spread being used to optimize said fitting of said curve .
7, step d. 7. The method of any one of claims 1-6, further comprising removing false positive guardrail structures.
8. Method according to any one of the preceding claims 1 to 7, wherein said tracking of said detected guardrail structure is performed using any suitable tracking algorithm or method, in particular a Kalman filter.
9. 9. A method according to any one of the preceding claims 1-8, wherein it is determined that the image frame contains two guardrail structures on different sides of the road and/or two curves are generated.
10. 10. The method of claim 9, wherein by examining the angular information of the guardrail structures, it is determined that the image frame contains two guardrail structures on different sides of a road.
11. 11. The method of any one of claims 1-10, further comprising the step of warning a driver when the driver is approaching the guardrail.
12. In a driver assistance system (1000) for detecting a guardrail structure,
a. a camera (1002) configured to capture a plurality of images;
b. a processor (1001) configured to be connected to the camera and configured to perform the operation according to any one of 1 to 11 above.
13. 13. The driver assistance system of claim 12, further comprising a radar, wherein the processor is configured to use information provided by the radar to generate guardrail curves.
14. 14. A vehicle comprising the driver assistance system according to 12 or 13 above.
15. 15. The vehicle according to 14 above, wherein the vehicle is a passenger car, a motorcycle, or a truck.
16. A computer-implemented machine learning method for providing a method for determining that an image frame contains a guardrail structure, comprising:
a. providing a labeled dataset of at least one image frame of guardrail structures in a database and at least one machine being a learning model;
b. processing the at least one image frame to extract angular gradient information of a guardrail structure of a guardrail;
c. and using the angular gradient information to train the learning model for classifying the image frames containing guardrail structures within the image frames.
17. 17. The method of claim 16, wherein the learning model is a support vector machine SVM or an adaptive boosting machine AdaBoost, or any suitable machine learning approach.
18. step b. 18. The method of claim 16 or 17, comprising implementing VODCA, a view-oriented, distributed, cluster-based approach to parallel computing.
19. A computer system (1000) comprising a memory (1003) and a processor (1001), wherein the processor is configured to perform the operations according to any one of 1 to 11 or 16 to 18 above. .

101 Guardrails T1-T6 Guardrail Structures 102 Roads S501 Provide Labeled Database for Guardrail Structures S502 Extract Angular Gradient Features S503 Classification Step 801 Drawn Curves 802 Roads S901 Input Frames S902 Apply Machine Learning Algorithm to Detect Guardrails S903 Cut Extract out angle information S904 Apply clustering S905 Extract non-vertical portion angle information S905 Consider non-vertical portion angle information and perform (polynomial) fitting 1000 Computer system 1001 Processor 1002 Camera

Claims (16)

A computer-implemented method for generating a guardrail pattern comprising :
a. capturing an image frame with a camera ;
b. determining that the image frame includes at least two guardrail structures having a non-vertical portion NVP and a vertical portion VP of guardrails, and using a clipping box to track the guardrail structures within the image frame ;
c. processing the cutout box information to extract angular slope information of one of the tracked guardrail structures ;
d. clustering the angular gradient information using a clustering method that produces clusters of information about angles of the clipping box components ;
e. extracting information about the non-vertical portion NVP of the clipping box ;
f. repeating steps c-e until all of the tracked guardrail structures have gone through steps c-e ;
g. and fitting a curve through said NVP of a tracked guardrail structure. The angular gradient information is an angular gradient description providing gradient lines of the vertical and non-vertical portions of a single structure of guardrails of the at least two guardrail structures and information about the angular gradients of the non-vertical portions relative to a reference. 2. Method according to claim 1, extracted by providing children, preferably gradient oriented histogram descriptors. 3. The method of claim 1 or 2, wherein the angular gradient information is extracted by implementing VODCA, a view-oriented, distributed, cluster-based approach to parallel computing. A method according to any preceding claim, wherein step g) comprises interpolating a curve through the detected guardrail structure to obtain a continuous curve indicative of the guardrail. A method according to any one of claims 1 to 4, wherein said curve is a polynomial curve through the intersections of sloped lines of NVPs of said at least two guardrail structures. step d. provides a mean value μ in each non-vertical portion and a corresponding spread σ , said spread being used to optimize said fitting of said curve. Method. step d. The method of any one of claims 1-6, further comprising removing false positive guardrail structures. Method according to any one of the preceding claims, wherein said tracking of said detected guardrail structure is performed using any suitable tracking algorithm or method, in particular a Kalman filter. Method according to any one of the preceding claims, wherein it is determined that the image frame contains two guardrail structures on different sides of the road and/or two curves are generated. 10. The method of claim 9, wherein by examining the angular information of the guardrail structures, it is determined that the image frame contains two guardrail structures on different sides of a road. A method according to any preceding claim, further comprising the step of warning a driver when he is approaching the guardrail. In a driver assistance system (1000) for detecting a guardrail structure ,
a. a camera (1002) configured to capture a plurality of images ;
b. a processor (1001) configured to be connected to the camera and configured to perform the operation of any one of claims 1-11. 13. The driver assistance system of Claim 12, further comprising a radar, wherein the processor is configured to use information provided by the radar to generate a guardrail curve. A vehicle comprising a driver assistance system according to claim 12 or 13. 15. The vehicle of claim 14, wherein the vehicle is a car, motorcycle, or truck. A computer system (1000) comprising a memory (1003) and a processor (1001), said processor being configured to perform the operations of any one of claims 1-11.

Images